Idea Transcript
Hacker News new | comments | show | ask | jobs | submit
login
Do Not Use Go for 32bit Development (abtinforouzandeh.com) 117 points by abtinf on Apr 8, 2012 | hide | past | web | favorite | 130 comments
jesstaa on Apr 8, 2012 [-]
The strategy the allocator uses is to reserve a large contiguous block of address space(not memory) on start up and then allocate actual memory from that block. On 32bit this is around 512MB, on 64bit it's something like 16TB. This makes allocating and garbage collection far simpler and it really shouldn't be a big ask for an operating system to give a process a contiguous block of address space and not interfere with it. But just like the pinning issue the smaller address space on 32bit means that things that expect to be at a certain address have a higher likelihood of getting in the way of the contiguous block. The fix for both this issue and the pinning issue is big changes to the garbage collector. Since the language and standard library have stabilised with Go1, changes to the runtime are going to be the focus of development so this issue will probably get a fix. xpaulbettsx on Apr 8, 2012 [-]
This is highly dependent on the video card and other PCI devices you are using, as they will often reserve huge chunks of VA space to map to device memory (and furthermore, require it to be in low addresses i.e. user space), making a contiguous reservation difficult. If you want to see exactly what's happening, SysInternals VMMap (http://technet.microsoft.com/en-us/sysinternals/dd535533?ppu...) will tell you exactly the VA space layout. Note that Linux has exactly the same problems with VA space, except they default to 3/1, instead of NT's default (but configurable) of 2/2. It seems that a quick fix to this issue would be to lower that number from 512MB to something like 128MB, though I know nothing about Go's internals. JoshTriplett on Apr 8, 2012 [-]
Getting 512MB of contiguous virtual address space does not require 512MB of contiguous free physical memory; contiguous virtual addresses can map to discontiguous physical memory. astrodust on Apr 9, 2012 [-]
That's what I thought. The address your user-space program uses and the actual address in memory are two different things, aren't they? Isn't this what enables the operating system to scramble the allocations it gives you to make it harder to implement a buffer-overflow attack? JoshTriplett on Apr 9, 2012 [-]
> That's what I thought. The address your user-space program uses and the actual address in memory are two different things, aren't they? Right. Userspace uses virtual addresses, which get mapped through page tables to become physical addresses. Some kernel addresses get identitymapped (meaning that the virtual address matches the physical address), while some kernel addresses go through translation as well. (Specifically, on Linux, kmalloc allocates identity-mapped addresses, while vmalloc allocates virtual addresses; vmalloc allows you to have a large virtually contiguous buffer without requiring a large physically contiguous region of free memory.) > Isn't this what enables the operating system to scramble the allocations it gives you to make it harder to implement a buffer-overflow attack? You can do Address Space Layout Randomization (ASLR) for either virtual or physical address spaces; you want to do it for whatever type of address an attacker could otherwise make use of. Userspace processes can use ASLR for their virtual address space, so that attackers can't make use of fixed addresses. The Linux kernel doesn't normally map virtual pages to any particular well-known location in physical address space, either. The kernel can also use ASLR for some (though not all) of its own kernel-space addresses. Beyond that, recent versions of Linux also try to avoid exposing kernel-space addresses to non-root users. Also note that ASLR doesn't generally introduce enough randomness in a 32-bit address space. 4ad on Apr 8, 2012 [-]
I/O devices are never mapped in user space in Windows, and I think that's true for other operating systems as well. xpaulbettsx on Apr 9, 2012 [-]
Ah, you're right, they have to be in low physical frames, I'm getting confused. nikcub on Apr 8, 2012 [-]
> On 32bit Windows machines, processes only have access to 2GB user space memory. You can change this in boot options to 3 or 4 (optimized) or enable PAE. see: http://msdn.microsoft.com/en-us/library/aa366778.aspx i'd try that before rewriting your code (also kill services you don't need, which is usually a lot of them. run -> msconfig). Edit: you also don't mention probably the most important piece of info: which version of windows you are running. Edit 2: make sure go is being treated as a background process Edit 3: it seems a lot of these tips should be in the Go docs for win32 - i'll add it when I have a moment georgemcbay on Apr 8, 2012 [-]
"Just set these boot options" is fine if you're just deploying this code to a server you control. I get the sense from various aspects of this guy's story that he's shipping executables to end users, which makes this "solution" a non-starter. You can't expect the user to go in and mess with these sorts of options just to run your program, and I wouldn't feel comfortable changing something like this automatically during install since it is such a fundamental OS setting (not to mention testing the installer against all versions of Windows would then become a nightmare you'd have to redo every release). malkia on Apr 8, 2012 [-]
Enabling /3GB and LARGEADDRESSAWARE might work, but I have found that because of certain video drivers (XP) would not work correctly with D3D or OpenGL apps (looks like some assumptions were made there that the space would be below 2GB). For one we could not get correctly our game and some of our tools to work on XP with /3GB (that was few years ago) Nowadays since we moved to Windows Vista/7 64-bit, we no longer have the problems (LARGEADDRESSAWARE) and our 32-bits sometimes go up to 3.5gb of usage. trimbo on Apr 8, 2012 [-]
Importantly, you will need to recompile with /LARGEADDRESSAWARE. 4ad on Apr 8, 2012 [-]
Yeah, that kernel flag is useless without stamping the binaries, and the Go linker doesn't know about this. barrkel on Apr 9, 2012 [-]
It's just a flag in the PE headers. It can be modified after the fact with a tool; the Go linker doesn't need to know. doug363 on Apr 9, 2012 [-]
But the reason why there is an explicit flag in the first place is that there's a lot of code out there that doesn't work with pointers >= 0x80000000, or when pointers could differ by more than that. Things like pointer "tagging", and signed integer overflow bugs, are examples of the sort of code that can be affected. Without knowing details about the Go code generation and runtime/garbage collection, it would be a bit risky to just set the flag and hope for the best. barrkel on Apr 9, 2012 [-]
Oh, I know; I've written bounds-checking logic for RTLs that use casts to integers and tests for negativity to check for overrun, implicitly assuming 31-bit address space. My only point is that a limitation of the Go linker does not preclude setting the flag. 4ad on Apr 9, 2012 [-]
No such problems in Go. I just checked and the flag is set by default in all Windows binaries. 4ad on Apr 9, 2012 [-]
> the Go linker doesn't know about this. I was wrong, it does, it's on by default :). xpaulbettsx on Apr 8, 2012 [-]
One thing to note, is that his problem is lack of contiguous VA space, not physical frames (i.e. "memory"). Killing services will do nada to help that, the only thing that will help is mapping less DLLs. 4ad on Apr 8, 2012 [-]
DLLs are loaded in a process address space only because they are referenced in the PE header or the process calls LoadLibrary(), I'm afraid killing processes won't cause mapping of less DLLs either. yuhong on Apr 8, 2012 [-]
It has nothing to do with PAE. Bootvis on Apr 8, 2012 [-]
It seems to me it does: With PAE Windows can use more memory. This extra memory can be used to allocate the 512mb chunck. Can you clarify? tankenmate on Apr 8, 2012 [-]
It's a VM (mapping) issue not a RAM (free memory) issue. 4ad on Apr 8, 2012 [-]
Also, with PAE, each process gets the same 32 address space as before. algolicious on Apr 8, 2012 [-]
At runtime init, Go programs attempt to reserve 512mb of RAM. If they can’t do this, they crash. Go needs the space because it is garbage collected - presumably, all GC languages, or any program that takes a similar approach to memory management, will be susceptible to the same problem. While it is true that most garbage collected environments will reserve some memory at startup, a hard minimum of 512 MB is not the norm. Do note that embedded Java has run on cell phones with minimal amounts of RAM for many years. _delirium on Apr 8, 2012 [-]
Fwiw, Go doesn't need a hard minimum of 512 MB physical memory, but it does seem to assume a contiguous 512 MB of virtual address space will be available. Since every process has 2 GB of virtual address space and grabbing the 512-MB chunk is the first thing Go does, this should almost always be safe, but on Windows it appears that can fail to be the case because in some configurations, DLLs are loaded during initialization and fragment the virtual address space. I can run Go fine on a 32-bit (Linux) VPS with 256 MB of RAM, though. ismarc on Apr 8, 2012 [-]
They only get scattered throughout when using cgo with the DLLs loaded by the Windows system by the application loader rather than by the go runtime. It appears to be very specific to how Windows loads DLLs at runtime. 4ad on Apr 8, 2012 [-]
Yes, though in this particular case the author told me cgo wasn't involved. This makes it very very peculiar. Usually when I see system DLLs (not 3rd party) rebasing it's either mallware in other processes that lazily loaded ntdll.dll and that had to be rebased or some legit program or mallware that does user mode hooking by injecting DLLs. It is NOT a Go problem, but Go could implement a workaround making the address space reservation in teh PE header. burgerbrain on Apr 9, 2012 [-]
Isn't Go normally statically linked if cgo isn't being used? Would that mean that the DLL's in the virtual address space definitely shouldn't be there? 4ad on Apr 9, 2012 [-]
Statically linked in Linux, in Windows the Go runtime and packages are statically linked into the binary, but the binary itself is dynamically linked to kernel32.dll because issuing syscalls directly is not supported. Why was kernel32.dll rebased when Go doesn't force the rebase is a very good question indeed. I suspect a 3rd party user mode hook, it doesn't even have to be malware, there are legit "security" and monitoring applications that use this technique. mbe on Apr 9, 2012 [-]
ASLR I would guess. 4ad on Apr 9, 2012 [-]
ASLR support needs to be stamped in the PE header. You can enable it globally, but the randomization space is smaller than it should matter for this. It's definitely something to investigate though. abtinf on Apr 8, 2012 [-]
You are correct - it was an error in my post. Go wants 512mb of virtual space. ismarc on Apr 8, 2012 [-]
Looking through the linked bug and the testing they did, the issue doesn't appear to be using Go for 32 bit systems. It's using 32 bit compiled Go programs that use cgo on 64 bit Windows systems. Understanding the why of the issue is extremely important in this case. Based on the discussion in the bug report, I would expect that using the 32 bit compiler for 32 bit architectures and 64 bit for 64 bit architectures would not result in the issue occurring. Before abandoning Go, I would very much identify that this is indeed the case. With Go's young age, I would very much use the architecture specific compilers for the specific targets which is different than normal expectations, but not at all unreasonable. abtinf on Apr 8, 2012 [-]
Unfortunately, this is not the case. 32bit compiled runs great on 64bit machines, where it gets the full 4gb of space and memory fragmentation at init is very unlikely to be a problem. Edit: similarly, 64bit also is immune because of how the main platforms allocate virtual address space to clients. ismarc on Apr 8, 2012 [-]
Hrm, that actually makes the issue tricky to solve. I don't have any Windows machines to work with, so I can't test, and I'm not very familiar with using Go on Windows in general, but is it possible to specify delayed load of the DLLs? That should allow the main Go system to get fully initialized (and allocate the chunk of memory it needs) before loading the DLLs needed for the cgo portions. 4ad on Apr 8, 2012 [-]
Nobody is using cgo, it's the system DLLs which get loaded in the middle of the address space. This is very very unusual, Windows tries hard to avoid such things. It could be solved by making the initial virtual address space reservation in the PE header, not requesting it in the initialization path, that would always work, even with cgo. sausagefeet on Apr 8, 2012 [-]
I don't believe this is true since the issue is that the GC is conservative so on a 32 bit system integers look a lot like pointers, so the GC won't free them. dchest on Apr 8, 2012 [-]
Did you read the post? sausagefeet on Apr 8, 2012 [-]
Nope, my bad. This is the 2nd or third post on Go not being any good on 32 bit environments in as many days. The other post I read is what my post is in reference to. dhconnelly on Apr 8, 2012 [-]
I'm a big fan of Go, but it would be nice to see a caveat about 32 bit systems on the Go website or blog. Or at least some official warning or notice that isn't just a mailing list discussion. malkia on Apr 8, 2012 [-]
If 512mb are needed as contiguous space, then one hacky solution is do define a BSS section of 512mb, or in "C" terms - global array char mem[512 * 1024 * 1024]; the executable loader would "allocate" this memory before loading any dlls, and later dlls that were supposed to be in that memory would be rebased Oh, and somehow Go would need to use this memory rather than allocate it. barrkel on Apr 9, 2012 [-]
This would, however, allocate memory rather than just address space. (Yes, it may be lazily committed, but it's still not as cheap as address space.) malkia on Apr 9, 2012 [-]
Once this is done, you can VirtualFree it (starting/ending from page boundaries), and let it back to the OS. Yes for some short amount of time, while it loads and goes to your code the memory is reserved. moldbug on Apr 8, 2012 [-]
I wouldn't call this "hacky" - it's exactly the right way to do it. For one thing, it makes the address of your arena constant. Unfortunately my experience is that a lot of C environments handle this very poorly. malkia on Apr 8, 2012 [-]
I called this hacky, as it would only work for executables, if this was in a dll/dylib/so shared library, then it might even make the things worse, as now even more stricter contiguous space would be neeeded. For example, if the language runtime was loaded as plugin or something. So one has to be aware that is not always the good solution for all purposes, hence hacky :) ExpiredLink on Apr 8, 2012 [-]
> I’m sitting here rewriting a ton of Go code in C. I don't believe a single word. He pretends to have written "a ton of Go code" without discovering the "real show stopper"? I'm not a Go fan (quite the contrary, Go isn't the step forward from C++ and Java I expected from Google). But this Anti-Go campaign few days after 1.0 smells. georgemcbay on Apr 8, 2012 [-]
I'm not the OP, but I've written quite a bit of Go code and I didn't realize this was an issue until it started blowing up on golang-nuts/HN/proggit the past couple of days. Until you actually have to deploy the code on end-user 32-bit systems, it is very easy to be blissfully unaware of this issue. And since Go1 "supports" Windows-x86, why would you assume anything other than that your code will "Just Work" on such a target? I don't think it is unreasonable to expect the Go team to give this issue more light. In my case it doesn't really matter, but if I were using Go to write end-user apps where 32-bit Windows installs were on the table, I could understand this guy's frustration pretty easily. eternalban on Apr 8, 2012 [-]
> Go isn't the step forward from C++ and Java I expected from Google Go is a lovely language, imo. Quirky in parts, but definitely a charmer, very fun to code in, and once you get the Go zen, simple and elegant code effortlessly follows. I haven't had this much fun since Java came out in mid 90s. Of course, I do agree with the other comments that would like to see Golang.org be more clear as to the platform coverage and the current state of Scheduler and MM -- but they are certainly upfront in the source tree (c.f. proc.c header comment), forums, and the issue tracker. So hacker fine print is there and available, and frankly (IMHO) that approach to disclosure may turn out to be a good filter for the growth of Go community. The issue at hand is nothing that time and resources will not solve -- they just ran out of time (c.f. RSC's comment before Abtin's in the issue.) pjmlp on Apr 8, 2012 [-]
>> Go isn't the step forward from C++ and Java I expected from Google > >Go is a lovely language, imo. Quirky in parts, but definitely a charmer, very fun to code in, and once you get the Go zen, simple and elegant code effortlessly follows. I haven't had this much fun since Java came out in mid 90s. Go is a step backwards in language abstractions, throwing away many of the abstractions that have become mainstream in the last decades. luriel on Apr 8, 2012 [-]
Sometimes to move forward you have to take a step back. Just because some abstractions have become mainstream doesn't mean they are good. I think precisely what many people love about Go is that it throws away lots of unnecessary abstractions and complexity that people have come to expect from languages this days. Go's approach is a breath of fresh air. wmf on Apr 8, 2012 [-]
So hacker fine print is there and available, and frankly (IMHO) that approach to disclosure may turn out to be a good filter for the growth of Go community. Creating bad PR by releasing "stable" code containing undocumented landmines sure is likely to affect the growth of the community. eternalban on Apr 9, 2012 [-]
Of course you are correct, but that is not exactly what Golang.org has done. I believe errors have been made on both end of this story. Please consider this: a version 1.0 of an open source project X has been released by a team that actively engages its user community. Release manager indicated before release (in issue tracker) that a special problem with platform P is not making the cut. It is true he didn't tweet it /g but it was a public announcement. Developer D has decided to use X1.0 to create a software product for paying customer C on said platform P on a tight production go live deadline. Apparently there were issues in his deployment testing regiment, as his initial tests were apparently satisfactory, and he typed a ton of code, and now he is busy retyping the same in C as something went boom with X 1.0 on platform P. IMO, it is not thoughtful to spec 1.0 software for your paying clients unless you know what you are doing and have land mine detectors and have reviewed field maps. (An example of folks fitting the requirement are Heroku -- they use Go in production.) IMO, stories like this will (a) stem the rush of those who compulsively (but superficially) seek 'the new' and then blog about their own misunderstandings; and likely keep thoughtful hackers who like to dig deep (and will avail themselves of the public resources and available channels) and make their own determination. And these hackers will build great software on Go and that will attract more developers. There is no real reason for Go to seek a footprint via "PR". PR for languages is for those who need users for their language's survial. Sun (RIP) didn't really need Java, did it? They had to push Java to gain users. Go has Google for its deployment foot print (as a client of sorts); is already an option in the cloud. Go has a solid team of engineers [1] behind it. It is definitely not a flavor of the month language and would not benefit from a flavor of the month community. All opinions, of course! [1]: for a sample of both: http://www.youtube.com/watch?v=HxaD_trXwRE abtinf on Apr 8, 2012 [-]
Believe what you want. The problem is intermittent and didn't appear on my test machines/VMs. trimbo on Apr 8, 2012 [-]
What's "a ton"? caf on Apr 9, 2012 [-]
This same Go implementation design earlier ran into some issues on Linux, too - in this case, with distributions setting conservative RLIMIT_AS limits. The technical details are interesting (https://groups.google.com/forum/#!msg/golang-dev/EpUlHQXWykg...) and Linus' response (http://lkml.indiana.edu/hypermail/linux/kernel/1102.1/00233....) is also instructive. aidenn0 on Apr 9, 2012 [-]
I had a similar issue with the default configuration of SBCL on a 64-bit vhost; the 64-bit version asks for 8gb of VAS and the vhost had a fairly small limit on VSIZE. Fortunately the heap allocation is configurable in SBCL; perhaps doing something like that on go would be the right thing? cjensen on Apr 8, 2012 [-]
In my opinion, the problem is that 32-bit Windows is intentionally broken by Microsoft. Unlike other 32-bit x86 operating systems, the consumer version of Windows refuses to addresses more than the first 4GB of physical memory, and on most machines a quarter to half of that space is consumed by PCI address space. Yes there are hacks around this. But the right way to deal with this is to just fold and accept what Microsoft is imposing: use a 64-bit version of Windows whenever possible and enjoy your memory. Once on a 64-bit OS, you can either write a 64-bit program, or you can write a 32-bit program and use a compiler option to get the full 32-bit address space. (Microsoft's compilers assume by default that you are an idiot who uses signed values for addresses and your program will break if they give you more than 2GB of virtual address space) The other lesson is don't use experimental languages you are uncertain of for actual product. yuhong on Apr 8, 2012 [-]
This has nothing to do with physical memory limits. Virtual address space != physical memory. pjmlp on Apr 8, 2012 [-]
The same also applies to other 32 bit OS, this is not Windows specific. tankenmate on Apr 8, 2012 [-]
Umm no, other OSes have different defaults for mapping and have different heuristics for how and where they map devices and libraries. Go on 32bit Linux rarely has the problems that the Windows versions do. 4ad on Apr 8, 2012 [-]
The PCI mappings are the same, either in Linux or Windows. I do agree that this particular report is a Windows problem, 512MB contiguous address space should be available at program initialization in every case. I do agree that Go on 32 bit Linux rarely has any problems, but this is true for Windows as well. What happened in the last few days is blown out of proportion. tankenmate on Apr 9, 2012 [-]
The _size_ of the mappings are probably the same, but there are very few PCI devices that _require_ to be mapped to a particular address, as opposed to a certain area of memory, i.e. under 4G or 1M. 4ad on Apr 9, 2012 [-]
Nothing is mapped under 1M in protected mode with paging enabled, that's user space. They still have to be mapped under 4G because 4G is all you got on 32 bit. cjensen on Apr 9, 2012 [-]
No. A 32-bit OS is capable, on the x86, of addressing much more than 4GB of physical memory. The 32-bit server versions of Windows, for example. 4ad on Apr 9, 2012 [-]
The address space of each process is still 4G. The reported problem and the one before it are related only to virtual memory, they have nothing to do with physical memory. PAE does not influence the address space of a process at all. fleitz on Apr 8, 2012 [-]
Somewhere in the source of Go will be the number 536870912, change this number to 134217728. Problem should largely be solved. Note: I'm not a Go dev, or have even programmed in it, but logically it should be a quick fix that you can try to save you rewriting your codebase. Given the importance of the number it's probably a #define or constant so should be pretty easy to find in one of the header files. 4ad on Apr 8, 2012 [-]
It's 512