Discussion: Openness to significant changes to the project?

[dkowis]

And by that I mean significant refactoring changes, not really changing functionality.

I'm using this library at work, and it suffers from a few problems:

• including a java logging library instead of only using slf4j-api
  • This is honestly a trivial fix. The slf4j-log4j library should be runtime scoped
• Uses an out of maintenance SSH library
  • I'd switch to sshj which is still receiving activity
  • Or instead, make it SSH library agnostic
  • I also want SSH-agent support, so that I don't have to do weird SSH RSA-only PEM format key things.
  • sshj supports modern algorithms and ssh key types
• doesn't support the optional JSON structure that the pyEZ library does
  • I'm not yet sure how to make this work, but I'd like to try :)

Not a problem, but a personal preference:

• I'd love to see it use gradle for the build system
  • I'm just a fan of gradle, as it gives us everything maven does, but also flexibility if we need it.
• github actions to do java matrix builds against 8, 11, and now 17
  • Would love to have some CI for all the java versions, to make sure it stays alive with modern JDKs
  • This is pretty trivial

How open is the project to some significant pull requests that change big things in the application?

What do folks think about the ideas I've listed here?

[GregDThomas]

Not a maintainer, but have had changes merged.

I’d agree with everything you’ve said.

I may even go further, and look at restructuring the API. Why is the Device class mutable, for example? Not sure the session belongs as a field of the class.

Can we cut down an all the knife-and-forking of XML and use a library instead? Probably an injection attack waiting to happen there.

My 2p.

[dkowis]

Can we cut down an all the knife-and-forking of XML and use a library instead? Probably an injection attack waiting to happen there.

I'm interested in what you're thinking here in more detail. Do you mean using a marshalling library like Jackson? Do you mean a stream based parser instead of documents and XPath?

For what I needed to do, I ended up implementing a STAX2 based xml parsing, because the configuration was pretty large. Jackson XML marshalling wouldn't cut it, due to the complexity and variety of the objects. All the objects had too many fields, and it was a mess to think about.

I also looked into using the Juniper XSD (which was an absolute nightmare), but it didn't code-generate (using XJC) into usable java objects that I could do anything with (everything was a generic type, without any specifics, so it was worse than parsing by hand)

I also looked into YANG (junipers yang), but couldn't find any good ways to take advantage of any code generation there, libraries are old and abandoned, and didn't cooperate on java 11, or just didn't work at all. Maybe spending more time understanding the YANG stuff could enable code generation of data classes and such. That would certainly help with parsing the output, and storing and then sifting through the data to build router automation stuff.

There's a lot of potential for code generation, especially for the data classes, but ultimately, I had to go back to manual STAX xml parsing to pull out the data.

From a pure pleasant development perspective, I'd love it if the XML generation used a kotlin DSL because manually creating a giant pile of XML is often very unpleasant using native java stuff. But this introduces a kotlin dependency that might not be wanted.

[GregDThomas]

I've not really thought out too much how the XML should be dealt with, just strong feeling that generating it by string manipulation is wrong.

I'd be wary of using generated code for the XML (YANG, XSD, etc.); IME the published schemas are more a guidance than anything else.

I've heard good things about Kotlin, but have had little chance to explore it properly.

The work I did do with XML manipulation (see classes in https://github.com/Juniper/netconf-java/tree/master/src/main/java/net/juniper/netconf/element) was the least required to get <hello> and <rpc-reply> going with a namespace prefix (my immediate problem at that time). It does this the "hard" way using manipulation of a org.w3c.dom.Document and an XPath library already in the classpath.

I think the only other XML element dealt with is the <rpc> message; it may be that Document + XPath will be enough given the limited number of packets we deal with - though I agree it feels a bit dirty.

[peterjhill]

One note on the ssh library, my use case requires a library that support FIPS compatible algorithms.
Initially the library used https://github.com/Juniper/netconf-java/tree/71f4ac92efa374b8d6c0b755fdddd9b111d96ff6/src/ch/ethz/ssh2

It was copied into the project, as at the time, it was not built using mvn or gradle or anything like that.

going with your idea "Or instead, make it SSH library agnostic", I could see potentially having an interface that the library used and having jsch and sshj implementations. I'd also make a fake ssh client implementation that could be used for unit testing. Perhaps the fake client could record the interactions, pass back fake data. Like a mock, but more powerful? (This could be a crazy useless idea).

I honestly request json back when possible. Our use case might be much simpler than yours. That's why I've never given that bit of code much attention.

It might make sense if when significant changes like this are made, we bump the version more than just a minor. Juniper has been pretty open to contributions.

[GregDThomas]

OK, so I had a look at this. After a bit of experimentation, I decided it was easier to start with a blank slate, copying over existing code piece-meal. When (!) it's finished, it should be easy to merge in to the existing repo, but in the meantime, it's standalone at https://github.com/GregDThomas/netconf-java3

It's still very much a work in progress. It will currently connect to a device using a username/password, and that's it. It doesn't even send a <hello>. But PRs of course welcome, as is other feedback. I'll continue to work on it as and when I have time.

Things to note;

• The Device class is now immutable. I don't see why a Device, once defined, needs to change. A lot of the code in the existing v2 Device class that relies on mutable fields simply proxies through to the session. My intent is for the caller to simply invoke methods on the session directly.
• There's an integration-test folder, too. Drop in YML files to src/integration-test/resources/secrets and it will use those details to connect to any devices you happen to have on your network as part of the tests. There's a sample YML file called template.yml in the parent folder.
• It uses an SSH library from Apache MINA (I've used MINA in past in other projects with success, albeit not the SSH library). Noting comments above about being SSH library agnostic, but baby steps!

[dkowis]

Mina is interesting, and probably a solid choice for an SSH client. Do you perhaps know how to use an ssh-agent provided key with it?

I know how to do it with sshj, and it's super useful to have the ssh key loaded from there, rather than directly in the code.

[GregDThomas]

Yes, MINA does offer the option to read the standard keys AIUI. Next on my list to look at.

[GregDThomas]

Yes, MINA does offer the option to read the standard keys AIUI. Next on my list to look at.

@dkowis I've updated https://github.com/GregDThomas/netconf-java3 so that it is now possible to connect with a password, supply a Base64 encoded private key, or possibly using the SSH keys stored in the .ssh directory under the user's home directory - unfortunately I'm unable to test this last option locally due to my setup. I'd appreciate it if you could give it a quick try to see if you can connect (create a .yml file in src/integration-test/resources/secrets based on the template at src/integration-test/resources/template.yml - simply remove both password and the privateKey entries and

.\gradlew integrationTest

should attempt to login using the default SSH keys.

Next up - send the <hello>!

[dkowis]

Work has taken me away from this stuff at the moment.

In my environment, the ssh keys aren't necessarily stored in ~/.ssh, but they are loaded with a compatible ssh-agent If the software could talk to the agent to get keys, that'd be ideal. I have done this before with sshj and another library: https://mvnrepository.com/artifact/com.jcraft/jsch.agentproxy.sshj/0.0.9 to allow bridging to that ssh-agent.

[GregDThomas]

A quick check of the MINA code suggests it may already work, or if not, it can be enabled to do so. The doc’s are unfortunately not clear, and it’s difficult for me to test. This is very much a background project for me, but will see what I can do.

[dkowis]

A quick check of the MINA code suggests it may already work, or if not, it can be enabled to do so. The doc’s are unfortunately not clear, and it’s difficult for me to test. This is very much a background project for me, but will see what I can do.

No worries, it's a side project for me as well! Thanks for tackling it! Don't stress yourself :D