On ibm’s website IBM DOCUMENTATION]
there is a phrase:
“Note: Big endian (BE) operating systems support 32-bit and 64-bit applications. Little endian (LE) operating systems exclusively support 64-bit applications”
Why does little endian operating systems not support 32-bit applications?
Is it because the power hardware doesn’t support it or the software isn’t adapted?
Does anyone know the specific reason?
If you look at the context this (web page) is about supported “Enterprise Linux” distributions and versions that run on IBM enterprise servers (POWER8/POWER9) and applies to the whole stack (including the kernel and hypervisor).
Early (year 2001) AIX and Linux (PowerMAC G4) ran 32-bit BE. By that year all emerging POWER systems were 64-bit and required 64-bit application support. ByARCH (32/64) support was an accommodation for old (enterprise) applications. At the cost of 2X build/test effort for the application runtime/libraries (20000-40000 Linux distro packages).
By the time LE (for Linux) was introduced (POWER8) enterprise applications were all 64-bit.
32-bit LE was only for embedded systems.
thank you very much.
At the risk of starting a heated debate, I’d like to get at least Steve’s opinion on what this means to the suggested addition of new (or changes to old) compliancy subsets.
The goal behind the existing subsets was to pragmatically leverage existing toolchain support as much as possible to support the spectrum of (then) foreseen processor implementations, potentially in the face of religious(?) concerns about such things as endian. So we’ve got the two low-end subsets requiring big endian to leverage existing embedded toolchain support, and 64b support being optional for them. Meanwhile we’ve got people asserting that was the wrong decision and that the whole thing should have been LE top to bottom, adding BE just for the AIX subset.
What do you see as the pros and cons of LE-based low end compliancy subsets?
Brad and Steve answer this very well, I wanted to cover from a different angle
if you look at Microwatt’s commit history from around 3 years ago you will see a patch from Paul Mackerras adding BE support. the total size of the patch was a mere EIGHTY lines of VHDL, along the lines of:
ldst_byte_order = MSR.LE
if instruction in "ldbrx, lhbrx, lwbrx, stbrx, sthrx, stwrx"
ldst_byte_order = !ldst_byte_order
that is literally all it takes to make a (dual LE/BE) bi-endian system in hardware, given that Power ISA 3.x has both the non-reverse and byte-reverse Load/Store instructions already (of course you need BR on I-Fetch too)
this startling hardware simplicity rather unfortunately masks the complexity and resources required to maintain 30,000 to 40,000 software packages, which Steven and Brad kindly referred to alrready
now, also worth pointing out is that you have looked at IBM’s website and assumed that it is the sole source of information and authority on Power ISA hardware products and services. IBM is the most important source of information, as the inventor, but not the only source of information, nor of Distros such as Debian.
for example you will find from Toshaan Bharvani’s company, Vantosh, that there is a POWER8/9 set of LE and BE 64-bit RHEL-compatible distros, called PowerEL.
https://www.powerel.org/ - see “Download LE” next to “Download BE”?
Vantosh do this work because their customers want it and pay them to do it! IBM is not maintaining this nor paying their employees to maintain it for IBM customers, it is Vantosh doing it for Vantosh customers 
and that really tells you everything you need to know: to maintain LE/BE, or 64/32, if there is not someone paying to do the work, then because it is such a lot of work it just does not get done!
very simple, really.
Toshaan kindly took some considerable time to explain this to me, in addition to the things I had already picked up about S.E Asia - China, Japan, India, Malaysia etc - still using VME Bus for 60-year-old Industrial Control systems that are so simple electronics that they are easy to maintain and repair and consequently do not need “replacing with something new” (VME Bus is BE).
The following simple statement is incredibly important:
Packet parsing of the IP Protocol is in BE order.
For banking, high-frequency trading, power-efficient routers and the gambling industry where they allow bets to be placed right down to the last millisecond, being able to parse TCP/IPand UDP/IP packets as fast as is possible is extremely important.
When the entire CPU is LE-only and the entire TCP/UDP stack has to be software-parsed before you can read or write it, it should be obvious that such software-parsing is completely unacceptable in such business environments.
what solution do they deploy because they are using Intel’s LE-only CPUs?
they use FPGAs or Custom ASICs to parse and perform full deep packet inspection in hardware, byte-reversing the entire stack up to ISO Layer 4 just so that the Intel CPUs do not have to do it in software!
This goes a long way to explain why lowly 500 mhz MIPS SoCs are still the de-facto standard in Consumer-grade and Industrial Routers, sold literally in the hundreds of millions quantities.
putting it plainly: whomever is saying “Power should not support BE” simply has no experience (i.e. no customers) in the Networking or Industrial Control Industry.
India’s Fast Breeder Nuclear Reactor for example is still using 50 year old 16 mhz Motorola 68000 CPUs, and VME Bus control systems for the control of the servos on the nuclear rods, and to read temperature, pressure and other sensors.
the absolute last thing in the world that they want there is to modify the software source code, inserting byte-reversing instructions right at the critical point of reading temperature or pressure sensor data and getting corrupted data due to a previously unforeseen 16/32 type-conversion error, or sending a command to the servos that smashes the uranium rods into the reactor at 256x the speed because they missed a byte-reverse Store on a 16-bit Memory-Mapped VME-Bus command!
India’s Nuclear Scientists and Engineers take at least 10 years to do a full “fail-safe” safety review and audit of any changes of components or systems in their Reactor. I got a chance to briefly speak to them in 2018 and advised them for god’s sake don’t use unleaded solder because unleaded solder grows whiskers over a 12-15 year period, causing short-circuits, and those VME Backplane Control boards are in service a lot longer than that.
This is an extreme case of a much more prevalent usage of VME (a BE memory and peripheral Bus) than many people realise, because it’s just not talked about in the West.
bottom line: cutting out BE from the Power ISA would be an absolutely catastrophic mistake.
remember: many customers and users cannot talk publicly about what they do and why they have picked whatever Hardware they have picked. it is therefore down to us to “Game-scenario”, requiring quite considerable empathy, to imagine what it is that they themselves cannot say.
Summary: NXP/Freescale. 32-bit. BE. Embedded. low-clock rate. Aerospace, Automotive, Industrial and Networking. FlexBus (hybrid VME / MCU8080). connect the dots.
except… that was then. now the market wants Secure IOT Edge AI running ZephyrOS or better Linux with an MMU and/or an IOMMU, at 1ghz and above in order to keep up but that is the only major addition to their requirements.
in the UK due to massive systematic hacking for over a decade of IOT webcam and router products with passwords “admin, admin” there will be legal requirements soon coming into effect (around one year) that an IOT Consumer/Industrial Product has to satisfy or it cannot be sold.
Europe will simply be adopting the extremely-carefully-drafted security-conscious documents near-verbatim except for translation.
Well it looks like the heated debate started without me 
Like anything “it’s complicated” and can only be less complicated by simplifying requirements.
First I cop a plea (in my advanced age) and state that there ain’t no such thing as ByArch (Bitness) or ByEndian at the application level.
The ABIs and runtime requirements are simply incompatible for both technical and historic reasons.
There are 4 (or six if you include AIX) software platforms and ABIs that sometimes coexist in pairs on the same kernel (Paul can share how much fun this is for the kernel).
At the application/process/library-runtime level they are mutually incompatible and so the application has to decide on the platform(s) and build for (that/those).
Because most Distro’s/Businesses can only afford to build 1 or 2 platforms (and after building 2 for a few years they want to get back to 1).
Applications of different Bitness or Endian coexist at the Network or Database API level. Shared file systems are possible within the same endian (As in Linux BE 64/32-bit). Shared memory or shared flat-file cross-endian is a nightmare. This is a problem even for X-server.
Now I will split the world into Enterprise and Embedded(IOT).
Enterprise requires consistency. Embedded tolerates inconsistency if that provides a clear (cost/performance) advantage.
Enterprise needs a large and consistent software ecosystem on which they can build value. They need this ecosystem to exist across the platforms their customers use. None of them want to custom build Boost, LaPack, MySQL … Because they have enough problems building their own applications.
That is why we have PPC64LE because the large enterprise application developers were having fits with endian within their own applications.
Enterprise also needs consistency as the architecture evolves. Their old code has to run on the new system. They will not use ISA features unless they are widely available across new and current hardware products. Subsetting is BAD! Retroactive subsetting is WORSE! This is why POWER5 and 970 blades were so confusing and POWER6 had to support VMX.
ISA evolution is a long game. It takes 6-10 years to get significant exploitation of new ISA features in the enterprise.
In my opinion a development laptop must support enterprise-level consistency with the application server.
Embedded Hardware and custom SDK systems are different. They can make their own choices/trade-offs as they are likely building their own software stack and development ecosystem. They can leverage the existing Linux/GNU software ecosystem to build their kit. But if they deviate too far from existing architecture/platform ecosystems the open-source community will be unwilling to accept the technical debt (upstream patches) associated with (custom) embedded platforms.
i’m so terribly sorry about that, steven
been there, had to face that, it embarrassed the sxxx out of a customer when i couldn’t configure exim4’s LMTP over a network between two machines (i’d tested x86-to-x86 LMTP over a network and it worked perfectly. deployment went really badly and had to be pulled).
the customer eventually informed me that it was because they had picked a BE OS (BSD) for the recipient machine, where the front-end recipients were LE. as nobody had ever done LMTP over a network between two such machines…
DCE/RPC on the other hand has a “Receiver Makes Right” bit, right in the frickin 18-byte header (i.e. every single RPC packet). the irony is that the only reason it was done that way is because “Sender Makes Right” was patented! but actually “Receiver Makes Right” is the sane way to do it. sender may simply “blat” (memcpy) data onto-the-wire and tell the receiver to re-order it, whereas “Sender Makes Right” requires a 3-way round-trip and that in turn makes stateless UDP-based RPC simply impossible (3-way round-trip ain’t in any sense of the word “stateless”).
so yes: application layer is where it’s easier to sort, there have been solutions for decades. but if you don’t care about interoperability it all goes to hell in a handbasket real fast, even at the application layer.
indeed. that’ll simply be down to fundamental lack of experience and basic computer science knowledge on their part. or is that unfair of me to say, given that my first FOSS network-related experience in 1994 was carrying out Network Reverse-Engineering for Samba (1.9.16), and Samba was forced to support some pretty wild hardware and some seriously intransigent c compilers, therefore i had to learn?
(edit, post-script…)
have you heard of datadraw? https://datadraw.sourceforge.net/
have you heard of datadraw? https://datadraw.sourceforge.net/
Have you heard of Shared Persistent Heap Data Environment
oooo, have now. so i remember it i’ve passed it on to the primary author of coriolis2 (a full RTL2GDSII VLSI tool). he described serious “global locking” difficulties to me a couple years ago that mean that 20 processes spend >95% of their time contending for a global lock (do the math there…) i would love to see a hybrid combination of LMDB, Datadraw and SPHDE for the insane (multi-gigabyte-resident RAM) tasks like VLSI routing.
we now resume our normal programme(?)
Where the dummy in the corner (I) come out of this so far is that our BE-based low end subsets have a good reason to exist, and that the merit of new LE-based low end subsets rides on the assumption that somebody will be building the stack and toolchain and that the technical debt will be acceptable, as described:
I guess the other possible conclusion is that the characterization of the new subset as embedded is “wrong”(?) and that there’s enough merit in the
that the worse-ness will be overlooked?
android, openembedded (aka yocto), gentoo, and so on, yes. or if you are extremely brave like Libre-SOC: 5+ months NLnet-funded work on rebuilding PowerEL (Fedora/RHEL) and Debian ppc64 and ppc64be for SFFS.
however at some point people go “hang on a minute, why am i as the OEM having all this crap, why are the NREs so high, why are my engineers struggling with all these ridiculous non-upstream-patches placing my product delivery schedule at risk? sod it i’ll go back to ARM”.
(100+ booths - not a single one with anything other than an ARM core):
as an aside: at that event i met Nordic RF (famous for BLE 32-bit ARM SoCs). we discussed and agreed that a reduced binary and data-usage size (automatically inherent from 32-bit binaries but also from upcoming SVP64) would be significantly advantageous for the reduction in power consumption and cost for the customer (smaller RAM, smaller NAND Flash) but what had not occurred to me before was that they have a huge number of customers doing ultra-expensive OTA updates, over 2G GSM, Satellite, at the end of 5+ km RS485 cables and so on.
in Embedded / AI / Edge / IoT, reduced binary size and reduced data usage is important.
If this addresses my question, I’m too dense to understand.
What was observed at the trade show was ARM’s business model in action. Provide the IP and a way to use it, and companies live within that to develop products.
IBM did the best it could given that we don’t have the same business model. We identified some cores and toolchains that were the best we could do to provide the bones of embedded and server products. We defined compliancy subsets around those two sets of bones. Meanwhile we have our own proprietary BE enterprise stack that people are also welcome to leverage.
The question was whether there’s justification for a new low end LE subset. A new subset requires the identified NRE. I’m not sure how to weigh LibreSoC’s getting funding for the development vs. the complaint about work being necessary to the more typical developer.
As far as code size is concerned, the straightforward and perhaps most effective solution is to bring back VLE to balance anyone using ARM’s thumb (if that still exists?). The incremental benefits of SVP64 won’t be something other traditional architectures will have an analog to.
Retroactive subsetting is WORSE!
This comment applies to Enterprise and any Development workstations for Enterprise.
Otherwise it is just a bad idea, even for Embedded. The problem with subsets is the toolchain components don’t get the testing that the current Enterprise platforms do. And as development continues “they forget” to maintain the special accommodations required for various embedded targets. Embedded toolchains get very fragile. Then it is all on the Embedded SDK developers to stay on top of the testing.
And it’s not just Embedded, it can be just being different. I remember the multiple times community developers broke IBM long double without knowing.
having seen this in action i’m not planning on doing a minor number of customers per SoC, i’m planning a massive long-term consistent subset that has huge (raspberry pi, arduino) Open Hardware takeup, compensating for exactly this type of “fire-and-forget” Board Support Package, of which Allwinner and other Chinese SoC Fabless Semis are the worst offenders (including blatant GPL violations that the Engineers know full well they are committing).
(in these “fire-and-forget” scenarios in China their primary customers are also “appliance OEMs” who want Electronics Goods to be sold alongside “spoons” or “wooly jumpers” because they heard from a mate that IPTVs or Tablets make better margins. the more enterprising Chinese companies actually set up a US shell company, ship vast quantity of cheap 7in Tablet product through customs, then once the last stock’s sold shut it down).
having witnessed this in action and seen one AU Engineer literally buy a stack of $35 China tablets a metre high none of which he could program and so had to consider creating a near-identical custom IoT product so expensive that nobody woud buy it, i know the scope of the problem, and that the solution is to “go arduino / rbpi / crowdfunded / pine64”.
TL Lim of pine64 sets a very strict policy that he only designs and sells hardware products that (a) a large number of pine64 community members have expressed an interest in and (b) use components that are 100% upstream in the linux kernel, u-boot, and so on.
the most extreme case of this phenomenon of the “FOSS Tail wagging the Corporate Dog” is the OLPC XO-1 CPU, the AMD Geode LX500, which 15 years later AMD still couldn’t get its customers to stop buying because the software support thanks to the OLPC was so insanely good (software downloads for XO-1s are still between 10-20,000 a month)
in other words, what you describe there Steven is the “vertical” embedded market where a very small number of engineers inside a company spend as little money on the Board Support Package associated with a given SoC as they can possibly get away with (remind me to relate the story about the quality of Marvell’s PXA “BSP” some time).
i’m planning something a leeetle more long-term, sustainable, and responsible.
