Rendered at 09:54:15 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
vc5 13 hours ago [-]
This takes me back 20+ years to when I had to create/parse EDI messages for various retailers when I worked at an independent record distributor.
I wrote a C library to parse the messages with callbacks in the style of Expat. There's binding for Perl to make it a bit easier to use. EDIFACT, X12 and Tradacoms are supported.
Anyways, unlikely anyone might need it these days, but there are some Perl scripts which can interpret the EDIFACT "directories" (basically message schemas), so you don't have to build your own message syntax definitions. It should be possible to convert the XML output that it generates into the Python needed to define message syntax, element definitions, etc.
Best of luck with it - there's a whole world of crazy industry standards to explore.
Oh, and consulting is a possibility if anyone needs it ;-)
Edison is a Python library for parsing EDIFACT messages. Instead of treating a message as a flat list of segments, it models messages as a syntax tree: segments can define groups of child segments, recursively.
You define the syntax tree once, then parse, validate, or pretty-print messages against that structure.
It works by converting the syntax tree to a state machine of possible transitions between segments.
nurettin 15 hours ago [-]
Once you know the tree, edi is trivial to parse. You can even do it with a loop and a state to store which level has which tag and what the current count is.
The real challenge was to discover what the tree was. Like wayfair order listings, cargo delivery messages, which are also similar to how Rhenus does them. Every vendor has their own format and standards.
twunde 14 hours ago [-]
For anyone who's encountered this, you likely have access to these but either they are being sent to random email alias at your company or you need to request these (clearinghouses typically have this, but you can also request from the vendor/partner). I'll never forget how _angry_ I was to find out after finishing a project that there was a spec with everything I needed and nobody had bothered to send it to me while I was implementing the project.
nurettin 31 minutes ago [-]
Once I started parsing example messages with the above method, the format became apparent. I even made a viewer for it which was useful because people we were integrating with were also making mistakes and I was the one correcting them thanks to the viewer.
The hardest part was europeans taking 15 day holidays every couple of months leaving work to someone else who had to start over.
Many may think that EDIFACT is easy to parse, which may be the case. But it is NOT easy to interpret.
There are hundreds of "substandards" per industry. For example for clothes or medicine, most vendors have custom fields outside of the standard which MUST be interpreted/sent or everything falls apart
Rygian 14 hours ago [-]
In my corner of the world, all official grammar descriptions finish with a sentence like "or another interpretation by bilateral agreement" (where bilateral means between any two vendors using the same "standard").
DonaldPShimoda 16 hours ago [-]
What is a "real" syntax tree? I wonder if the intent was "concrete" (the contrast to an abstract syntax tree), or if it's meant colloquially (as in "you really get a syntax tree" or "you get an actual syntax tree") rather than as a technical term.
warpspin 15 hours ago [-]
I think OP means that it will parse EDIFACT with the tree structure it actually has in its specification, instead of a simple list of messages like many people do.
EDIFACT normally has (possibly repeating, possibly nesting) segment groups and a lot of code trying to handle EDIFACT skips over that.
twunde 14 hours ago [-]
Surprisingly this problem of treating EDI files as lists instead of the tree structure it actually is, is widespread even among clearinghouses. In retail energy, clearinghouses would often support their own custom format, which typically was either CSV or Excel. If you used that, you would routinely run into data integrity issues.
At this point, I'm pretty convinced that if you're running into EDI (or its sister formats like HL7) processing issues, its because you've got the wrong data structure in place or you aren't keeping your rules up to date. Once you've got the right architecture in place, you can easily manage 100s of provider/partner-specific rules with one person at 20% time. If you don't your whole company is going to be dealing with data integrity issues forever
_1tan 13 hours ago [-]
Care to share what "right architecture" could look like? There's surprisingly little public engineering reading about EDI...
vc5 12 hours ago [-]
Mentioned elsewhere in the thread, but it's possible to parse the (mostly) machine readable specs for EDIFACT messages to create schemas for reading massages.
A library that I wrote decades ago has some Perl which can do that to produce XML definitions from the UN/EDIFACT published directories. I hope that it can be useful to others: https://github.com/davidcoles/medici
LouDNL 15 hours ago [-]
Really nice! I've been preparing to parse this for a possible customer in another language. Thanks for the insights.
I wrote a C library to parse the messages with callbacks in the style of Expat. There's binding for Perl to make it a bit easier to use. EDIFACT, X12 and Tradacoms are supported.
Anyways, unlikely anyone might need it these days, but there are some Perl scripts which can interpret the EDIFACT "directories" (basically message schemas), so you don't have to build your own message syntax definitions. It should be possible to convert the XML output that it generates into the Python needed to define message syntax, element definitions, etc.
Best of luck with it - there's a whole world of crazy industry standards to explore.
Oh, and consulting is a possibility if anyone needs it ;-)
-- https://github.com/davidcoles/medici
You define the syntax tree once, then parse, validate, or pretty-print messages against that structure. It works by converting the syntax tree to a state machine of possible transitions between segments.
The real challenge was to discover what the tree was. Like wayfair order listings, cargo delivery messages, which are also similar to how Rhenus does them. Every vendor has their own format and standards.
The hardest part was europeans taking 15 day holidays every couple of months leaving work to someone else who had to start over.
There are hundreds of "substandards" per industry. For example for clothes or medicine, most vendors have custom fields outside of the standard which MUST be interpreted/sent or everything falls apart
EDIFACT normally has (possibly repeating, possibly nesting) segment groups and a lot of code trying to handle EDIFACT skips over that.
At this point, I'm pretty convinced that if you're running into EDI (or its sister formats like HL7) processing issues, its because you've got the wrong data structure in place or you aren't keeping your rules up to date. Once you've got the right architecture in place, you can easily manage 100s of provider/partner-specific rules with one person at 20% time. If you don't your whole company is going to be dealing with data integrity issues forever
A library that I wrote decades ago has some Perl which can do that to produce XML definitions from the UN/EDIFACT published directories. I hope that it can be useful to others: https://github.com/davidcoles/medici