I am working on an application that has SDKs in multiple languages. Currently Java, JavaScript, Dart, and Go, but ultimately we’d like to have an SDK for every major language. Our primary test suites are written in Go, which means our other SDKs are not well tested. I do not want to write or maintain test suites in four or ten different languages.
What I would like to do is choose a language to write the tests in, define a test harness interface, implement that test harness for each SDK, and write the tests using that harness. Of course I could do this with RPC/HTTP/etc but that would add significant complexity. I’d prefer to write the tests in a language that has a meaningful degree of interop/FFI with most of the major languages. Lua comes to mind, since it seems like someone has built a Lua interpreter for basically every language in existence, but I have very little Lua experience and I have no idea how painful it might be to do this in Lua. I am open to other suggestions besides interop/FFI and RPC, though I don’t want to take the approach of creating test templates and generating the tests in each language. I’ve done things like that and they’re a pain to maintain.
I don’t think you really have a choice TBH. Trying to do something like that sounds like a world of pain, and a bunch of unidiomatic code. If you can’t actually support 4 to 10 languages, maybe you should cut back on which ones you support?
One interesting thing you could try if you really don’t want to cut back is to try having using an LLM to take your officially supported code and transliterate it to other languages. I haven’t tried it at this scale yet, but LLMs are generally pretty good at tasks like that. I suspect that would work better than whatever templating approach you’ve used before.
If neither of those approaches works, everything speaks C FFI, and Rust is a modern language that would work well for presenting a C FFI that the other languages can use. You’re probably not hot on the idea of rewriting your Go tests into another language, but I think that’s your only real option then.
I don’t think you really have a choice TBH. Trying to do something like that sounds like a world of pain, and a bunch of unidiomatic code. If you can’t actually support 4 to 10 languages, maybe you should cut back on which ones you support?
To be clear, the SDKs themselves are hand-written; I’m not trying to do anything fancy there. In terms of designing and writing the SDKs, we can manage that for the 4 we have now. The issue is testing. The main system is a collection of services that are accessed via an API. That API can be accessed directly through function calls, or via HTTP or RPC. Our integration tests interact with the system through that API. The SDKs have a moderate amount of logic so they’re not simple HTTP/RPC clients, but maintaining multiple (idiomatic) versions of that logic is not too much of a burden. The issue is that I want a single test corpus that I can use to validate each SDK without having to rewrite that test corpus in each language. Ideally I’d like the integration tests to be that test corpus.
If neither of those approaches works, everything speaks C FFI, and Rust is a modern language that would work well for presenting a C FFI that the other languages can use. You’re probably not hot on the idea of rewriting your Go tests into another language, but I think that’s your only real option then.
I was assuming I’d need to rewrite my tests in Go given that Go’s FFI support for anything other than C is not somewhere I want to go again. I have been meaning to learn Rust so I might just do that.
How complex are the API calls you need to make? Debugging interop mismatch can end up being more work than writing the tests if you need to deal with complex types.
Yeah, that’s pretty much what I was thinking too. The combination of a c API and a JVM API (and maybe .NET if you’re in Microsoft land?) Hits most FFI available in languages I’ve seen. I can’t think of any language I’ve used that couldn’t Interop with either a c library (.a or .so) or JVM library (.jar). However I’ve never used any .NET system seriously, so I don’t know about them.
FWIW I regularly remake the same API based game whenever I start a new job working in a new environment to test that my environment is “up to snuff” with my development methodologies. I’ve never needed to port more than API.a and API.jar to play around in any language. I’ve ported that system to at least 100 languages over the years, and while some have more friction than others, and often the c/JVM paradigm doesn’t line up well with the target language, it is always effective.
Would be interested to hear more about this game. How long does this take during work time, the evaluation criteria you hit on (is this testing your IDE setup, your knowledge of build tools, features of the tech stack like threading, etc etc)
The .NET languages support c interop. The documentation could use some love though.
How much does your SDK do? If it’s just wrapping calls to an HTTP API, use something like OpenAPI / Swagger to document the API, then auto-generate client libraries based on the OpenAPI specs.
Then if you add any language-specific niceties on top of the auto-generated code (i.e. accessor functions to set up user credentials etc) you have to write tests for those parts in that particular language. But the bulk of the API you can test in whichever language you prefer, then just assume the code generator is doing its job and creating a compatible API in the other languages.
Part of it is an HTTP/RPC interface, but that’s not the party I want to test.
