Can't "su" to non-superuser : Permission denied

[ttcoder]

~/Desktop> useradd tts_guest
~/Desktop> su tts_guest
su: /bin/sh: Permission denied

Used to work in earlier PM builds, e.g. 48168 Stopped working for a while Worked again as recently as 495xx No longer works since a few weeks, including in this fresh install of hrev49627

Previous times, I could find an explanation (ll /bin or ll /system/lib was showing wrong read permissions, some libraries or bash itself were not world-readable and world-executable e.g.) but this time I can't see the reason for the "permission denied".. Might be something even more obscure than /system/lib :-)

Originally stumbled on this as I wanted to be able to ssh into client stations as a non-priviledged user, but luckily the problem is reproducible also by simply using the su command so let's focus on that.

[axeld]

The strace output shows that it fails to open "/bin/sh", ie. that's the call that returns that error:

_kern_open(0xffffffff, "/bin/sh", 0x0, 0x0) = 0x80000002

[ttcoder]

And same if configuring the user directly on "bash" (instead of the "sh" symbolic link):

[  1401] _kern_open(0xffffffff, "/bin/bash", 0x0, 0x0) = 0x80000002 (16 us)

Oddly, the strace output differs quite much when invoking su user (i.e. when you hop into yourself), su seems the ignore the contents of /etc/passwd:

[  3124] _kern_read_stat(0xffffffff, "./bash", true, 0x7187f160, 0x58) = 0x80006003 No such file or directory (13 us)
[  3124] _kern_read_stat(0xffffffff, "/boot/home/config/non-packaged/bin/bash", true, 0x7187f160, 0x58) = 0x80006003 No such file or directory (17 us)
[  3124] _kern_read_stat(0xffffffff, "/boot/home/config/bin/bash", true, 0x7187f160, 0x58) = 0x80006003 No such file or directory (7 us)
[  3124] _kern_read_stat(0xffffffff, "/boot/system/non-packaged/bin/bash", true, 0x7187f160, 0x58) = 0x80006003 No such file or directory (12 us)
[  3124] _kern_read_stat(0xffffffff, "/bin/bash", true, 0x7187f160, 0x58) = 0x00000000 No error (11 us)
[  3124] _kern_read_stat(0xffffffff, "/bin/bash", true, 0x7187f190, 0x58) = 0x00000000 No error (6 us)

I tried to look at lib dependancies (strings < /bin/bash | grep lib) a bit but they seem to have their world-readable bit set.. (and if access to a library was the culprit, the strace output would hint at a library and not bash.. Or is that dependant on runtime_loader 's behavior ?)

[korli]

I tried locally and I had to

chmod 666 /etc/profile

to avoid a "Permission denied" message. Hope this helps.

[ttcoder]

Might be something different between our systems, korli:

~/> chmod 666 /etc/profile  # also tried 777 (incl. x bit)
~/> ll /etc/profile
-rw-rw-rw- 1 user root 2243 Jul 30 02:40 /etc/profile
~/> su tts_guest
su: /bin/bash: Permission denied

Looking again at /system/lib I see that the x86/ subfolder indeed has faulty rights (500 instead of 555), but bash-4.3.42-1-x86_gcc2.hpkg theoritically would not access x86 libs anyway IIUC

Another odd test idea I've had: making a bfs copy of the packagefs bash (only bash itself, not its libs) into /tmp, and using that -- turns out it yields a different error, "killed by death":

~/> useradd -s /tmp/New\ folder/bash tts_guest
~/> su tts_guest
Killed (by death)

strace:

[ 20211] _kern_setreuid(0x3e8, 0xffffffff, true) = 0x00000000 No error (0 us)
[ 20211] _kern_open(0xffffffff, "/tmp/New folder/bash", 0x0, 0x0) = 0x00000003 (40 us)
[ 20211] _kern_access(0xffffffff, "/tmp/New folder/bash", 0x1, true) = 0x00000000 No error (10 us)
[ 20211] _kern_read(0x3, 0x0, 0x71cfc548, 0x100) = 0x00000100 (6 us)
[ 20211] _kern_close(0x3) = 0x00000000 No error (4 us)
[ 20211] _kern_open(0xffffffff, "/tmp/New folder/bash", 0x0, 0x0) = 0x00000003 (10 us)
[ 20211] _kern_stat_attr(0x3, "SYS:ENV", 0x71cfc9e8) = 0x80006003 No such file or directory (11 us)
[ 20211] _kern_close(0x3) = 0x00000000 No error (2 us)

EDIT: maybe bash attempts to open itself (wow!?) à la B_READ_WRITE, which fails in packagefs (regardless of whether or not the 'w' bit it set or not, of course, being a read-only fs), but succeeds in bfs, allowing the bfs variant to get past that error and hit another one downstream ?

EDIT2: yea also tried changing rights to config/settings/bash_history, allocating its own homedir instead of /boot/home, makes no difference; strange as that is, it seems the bash file itself is the issue. Running ll /system is a bit scary by the way: most of its contents is 600 (instead of 644) ! Including runtime_loader... I wonder if VFS/BFS will allow non-superuser access to runtime_loader if it's rw-------" ?

[ttcoder]

That one probably deserves its own e-Mail rather than an edit, sorry :-) Here goes again:

Running ll /system is a bit scary by the way: most of its contents is 600 (instead of 644) ! Including runtime_loader... I wonder if VFS/BFS will allow non-superuser access to it, if runtime_loader is rw-------" ?

[axeld]

If it works differently for korli, then this is likely a build issue. I could imagine that the build process doesn't take care of manually setting the umask, and just adopts the setting of the host, causing improper access rights. Having said that, that's just a theory, I didn't look into it at all yet.

[ttcoder]

Tried to override runtime_loader with a custom package, now it's -r-xr-xr-x 1 user root 154316 Sep 15 13:43 runtime_loader but despite that I still get _kern_open(0xffffffff, "/bin/sh", 0x0, 0x0) = 0x80000002 , so I guess there is another system file involved when loading /bin/bash

@axel That sounds very possible, that hypothesis would explain why I get a system folder as "r-x------" sometimes, and as "r-xr-xr-x" at other times: it'd be because it depends on which of the build-bots I get my packages from when downloading a nightly or pkgman up.

FWIW, in the past few weeks I've reviewed my jamfiles/JamRules and added MODE on xyz.. = 755 so that all my apps are built with the right flags, as I realized they were not (don't remember if it was caused by the umask or if Jam actually enforced a MODE-on variable but that variable had the wrong value).

If the Haiku build process actually leaves room for umask (and does not have a "MODE on xyz" in jamrules) then that'd be another way to fix this yes. I hesitate to put this on bonefish's shoulders as he has lots on his plate already, but that'd be right up his alley I guess 8-)

[bonefish]

Just built a hrev49645 gcc 2 hybrid (on openSUSE 13.2). Can't reproduce the issue. packagefs's open() hook uses a standard VFS function to determine whether access should be allowed. It is fed the same file mode *stat() returns (i.e. what ls -l shows). So, if the file's permissions allow read access, opening the file should work fine. At least as long as its path can be traversed, i.e. the ancestor directories have executable permission.

From just your failing syscall it is not possible to tell much. Please enable debugging in packagefs:

SetConfigVar DEBUG : HAIKU_TOP src add-ons kernel file_systems packagefs : 1
	: global ;

It will badly spam your syslog, but hopefully we'll see which packagefs hook fails exactly (if any).

Regarding directory permissions: AFAIK ATM the build system uses the default permissions in any case it creates a directory and the package tool just stores the permissions it finds. So, indeed, if you have a restrictive umask, the packages you build will inherit those restrictive permissions. I guess that's something we might want to address (or at least document).

I guess a fairly simple solution would be in build_haiku_package to update all permissions ​before invoking package. Alternatively the package tool could grow an option to copy the owner permissions to group and others (or rather one for preventing that new default).

[ttcoder]

Looks like I need help with tracing:

After adding SetConfigVar DEBUG ... in UserBuildConfig, building packagefs from scratch, installing and confirming the install I did not get extra output to syslog nor the /var/log/packagefs.log file mentionned in the header; by hacking around I realized the debug macros were not activated at all, so I edited the DebugSupport.h header and added #define DEBUG_PRINT 1 and there the macros are compiled in, but there's still no output.

*

Anyway I've been digging in the jamfiles and found something interesting:

The Application, Preference ..etc rules all end up invoking the Link rule, which does have a "MODE on..." variable and invokes ChMod with 755 (rwxr-xr-x) , as seen here:

​http://cgit.haiku-os.org/haiku/tree/build/jam/OverriddenJamRules#n35

So the Application > Main > MainFromObjects > Link rule chain works in that respect.

​Shared libraries (.so) end up calling Link too:

SharedLibrary > SharedLibraryFromObjects > MainFromObjects > Link

One that does NOT is runtime_loader, because its ​jamfile is special: it makes "raw" calls to..

• Objects
• Ld

and that's it, so it never gets a chance to call ChMod and set a MODE value for it. Seeing as it is special case linking-wise, maybe the ChMod + MODE should be special-cased and added "raw" to its Jamfile as well ?

Finally, and I don't know how I missed that before, the reason the libraries are not accessible is.. The lib/ directory itself is 700 !

~/> ll /system/
...
dr-x------ 1 user root       0 Sep  5 22:03 lib

Are "generated" folders made with the MkDir jam rule ?

I'd test my theory, but I don't know if I can override a directory "as-is" rather than files inside.. So can't test my theory by crafting a custom package, but that looks very much like the culprit here.

Anyway, is that the right diagnosis ?

[ttcoder]

Yay!!

After crafting a package that overrides..

• /system/bin
• /system/lib
• runtime_loader

I get this :-)

~/Desktop> su tts_guest
/boot/home/Desktop> id
uid=1000(tts_guest) gid=100(users) groups=100(users)

[bonefish]

Replying to ttcoder:

After adding SetConfigVar DEBUG ... in UserBuildConfig, building packagefs from scratch, installing and confirming the install I did not get extra output to syslog nor the /var/log/packagefs.log file mentionned in the header; by hacking around I realized the debug macros were not activated at all, so I edited the DebugSupport.h header and added #define DEBUG_PRINT 1 and there the macros are compiled in, but there's still no output.

I copied the SetConfigVar line directly from my UserBuildConfig after verifying that it works. Possible errors are: 1. A syntax error in your UserBuildConfig. Depending on the error it may or may not result in an error message while building; jam will not abort, however. 2. You added some override of the DEBUG variable in some other place. I guess you'd know (or git status would tell). 3. Wrong jam being used. Though I suppose only Haiku's jam is still able to build Haiku at all, so that's unlikely.

Adding the suggested line in UserBuildConfig has the same effect as adding DEBUG = 1 ; in packagefs's Jamfile. The latter is just discouraged.

You can see whether something is built with DEBUG = 1 already while building. The generated files are placed in a different location in your "generated" directory ("objects/haiku/x86_gcc2/debug_1/..." instead of "objects/haiku/x86_gcc2/release/...").

Anyway I've been digging in the jamfiles and found something interesting:

The Application, Preference ..etc rules all end up invoking the Link rule, which does have a "MODE on..." variable and invokes ChMod with 755 (rwxr-xr-x) , as seen here:

​http://cgit.haiku-os.org/haiku/tree/build/jam/OverriddenJamRules#n35

So the Application > Main > MainFromObjects > Link rule chain works in that respect.

​Shared libraries (.so) end up calling Link too:

SharedLibrary > SharedLibraryFromObjects > MainFromObjects > Link

One that does NOT is runtime_loader, because its ​jamfile is special: it makes "raw" calls to..

• Objects
• Ld

and that's it, so it never gets a chance to call ChMod and set a MODE value for it. Seeing as it is special case linking-wise, maybe the ChMod + MODE should be special-cased and added "raw" to its Jamfile as well ?

This could certainly be done, but there are probably plenty of other (non-executable) files that are generated without explicitly setting permissions. And in the end they need to be correct in the packages/on the image rather than in the "generated" directory. Since you probably use a restrictive umask for some reason, circumventing it doesn't seem right.

Finally, and I don't know how I missed that before, the reason the libraries are not accessible is.. The lib/ directory itself is 700 !

~/> ll /system/
...
dr-x------ 1 user root       0 Sep  5 22:03 lib

Are "generated" folders made with the MkDir jam rule ?

If you mean the directories in the build's "generated" directory, most of them are. The package contents is, however, assembled by the build_haiku_package script and its sourced (generated) scripts. It uses plain mkdir.

Anyway, is that the right diagnosis ?

I guess so. The /system/lib directory however cannot be to blame, since that isn't where the bash is located. But as you mention /system/bin in your latest comment, that certainly can be.

[ttcoder]

IIUC the umask that's relevant is the one on the build-bot that created the Haiku image used by me and Dane ? That's probably what you referred to bonefish, just re-wording to be sure we are apples-to-apples.

Anyway I can make preparations to put my package "hack" in production, in case there are obstacles to solving this ticket.. The directories are easy to override without side effects, but overriding runtime_loader means it's "cast in stone" and our stations will miss potential updates to it though, so if there could be an interim fix (waiting for the big all-encompassing fix) addressing just that one, that'd be great :-).

Regarding packagefs build, I retrieve it from /Haiku3/_develop_hrev/haiku/generated/objects/haiku/x86_gcc2/debug_1/add-ons/kernel/file_systems/packagefs

======

Dependancy wise, confirming, I need all 3 elements here. Each missing one triggers an error:

• missing access to lib/ : runtime_loader: Cannot open file libreadline.so.6: No such file or directory
• missing access to bin/: Permission denied
• missing access to runtime_loader: Killed (by death)

Only if my hack package does contain all three, I can su to tt_sguest

====

My 2 cents on rwx policy:

The process of building Haiku should be self-contained, directed by Jamfiles directives, rather than environmental values like umask that are all over the place

That should be configurable (by e.g. changing the value of $(EXEMODE) & friends in UserBuildConfig) for those that need custom tailored builds, but other than that, people should have an assurance that if they use an out-of-the-box git clone of the Haiku repository, they can build a reproducible, apples-to-apples Haiku image on any computer they have, without mysteries.

If you look at OpenSSH for example, it's software that has a very precise idea of what each file's permission should be: the private RSA keys must not be group-readable and world-readable, regardless of the umask. They likely have parts of their makefiles or build scripts that hardcode access to certain files in their "packages", simply because in some cases those permissions serve a different purpose than umask (which serves a purpose for day-to-day tasks, creating new files, not building packages or operating systems). Other examples include executables that are "special", like su, with the 's' bit to "inherit-root-privileges" (or conversely, "do *not* inherit privileges"). If their makefiles/jamfiles relied on the umask to get the correct right that would be a security risk. Rights are (rightly) hardcoded for these. Deviations from these hard-codings should be like changing the source code in .cpp files : something to do for tinkerers and tweakers, not left to such general states as the general umask IMHO. Might be missing something though, as my perspective is colored by my being an app developer, not an OS developer..

EDIT: the most use of umask I could imagine, and even that is a stretch, is if the configure script had a smartty-pants detector that went, "I see that you are about to build Haiku from a machine that has such and such umask, do you want to create the Haiku build with an /etc/profile that invokes the umask command with that same value?". -- But even that is a stretch of an enhancement: if somebody wants to build a Haiku image with a customized umask in said image's /etc/profile, there are better ways to do that.. But yea, the generated /etc/profile is the most I can imagine having to do with the umask, the system files should be the responsibility of the build system IMHO.

[pulkomandy]

While the generated Haiku image should not change depending on the umask, we don't want to force the files in the generated dir to have unrestricted permissions. If the user doing the build set the umask to a restrictive value, there is probably a reason to do so. This means, we should leave the files as they are in the generated directory, but then "fix" the permissions on the fly as we add them to the packages/image. This way, the umask setting on the host is honored, and still we get a working Haiku image in the end.

In the case of the build bots, IIRC buildbot itself sets a restrictive mask by default, we had noticed this and set a less restrictive one in our configuration files, but maybe not all bots have been configured the same way.

[ttcoder]

Oh ok.. I see what bonefish meant now. So it'd mean going the other way around, removing the MODE-on and ChMod invokations from the Link rule, once there is a downstream system in place to set the correct rights on all relevant files in the packagefs realm (and in the bfs realm maybe too: /system/settings/ ..etc, not sure). That makes sense. Tough it might not be completely trivial for you guys to implement: currently the "knowledge" is nicely refactored upstream, as it is/was in per-file jam rules (e.g. the rule in charge of linking an executable "knows" that the executable must have the x bit), by opposed to moving that knowledge downstream to a more general script that looks at each package as a whole.

I'll post back some info about the build bot with restrictive umask that created the image I'm using, from the ReadMe.md file that was packed with the image. I think that bot was named something like "geist-4-bot" or some such.

[ttcoder]

There it is for the laptop -- this should be accurate as I did not yet pkgman-update from that:

Build details
-------------
Haiku repository: hrev49627 <https://git.haiku-os.org/haiku>
(..)
Configure command: ../../haiku/configure --use-gcc-pipe --distro-compatibility official --use-xattr-ref -j2 --cross-tools-prefix ../../cross-tools-x86_gcc2/cross-tools-x86_gcc2/bin/i586-pc-haiku- --cross-tools-prefix ../../cross-tools-x86/cross-tools-x86/bin/i586-pc-haiku-
Jam command: jam @nightly-raw
Host machine: geist-bot4-ubuntu64 [Linux hmini3 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11 14:29:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux]

As to the tower/desktop with the same symptoms, it originates from a nightly by arfonzo-bot1-debian but was updated since, so not sure whose umask is at fault there.

[ttcoder]

So here's the two-liner change we need, in case someone wants to make life easier for us without having to re-read the whole ticket history above ;-)

/Haiku3/develop_hrev/haiku/src/system/runtime_loader> git diff .
diff --git a/src/system/runtime_loader/Jamfile b/src/system/runtime_loader/Jamfile
index 3a19cf1..0699fe8 100644
--- a/src/system/runtime_loader/Jamfile
+++ b/src/system/runtime_loader/Jamfile
@@ -95,6 +95,9 @@ local sources =
 
 Objects $(sources) ;
 
+MODE on runtime_loader = $(EXEMODE) ;
+Chmod runtime_loader ;
+
 Ld runtime_loader :
        [ FGristFiles $(sources:S=$(SUFOBJ)) ]
        libruntime_loader.a

(i.e. apples to apples with ​this so things are cleanly consistent)

(I checked with jam -n and it does produce a good-looking output: chmod "755" ../../../generated/objects/haiku/x86_gcc2/release/system/runtime_loader/runtime_loader" )

Alternatively, there was talk above of fixing the umask on the build-bots, or going with an overhaul of the permissions enforcement. Would not hurt to apply the above two-liner consistency clean-up/fix while waiting for those, though

[ttcoder]

ping

Dane has expressed interest in using a newer hrev in production but I'm veto-ing that until I get that gosh-dang Chmod runtime_loader ; two-liner commited to the Haiku repository :-)

Who do I need to send a box of chocolates to ?

[dsuden]

Is there someone who can help us out with this? Would sure appreciate it!

[Barrett]

Fixed in hrev50187.

Next time just make more evident there's a patch :-)

[anevilyak]

This change breaks the x86_64 build.

[ttcoder]

@Barrett please note (it's not obvious given this ticket's length indeed!) that the two-liner is not an actual fix for this ticket, just a temporary work-around so that we at tts can move to a newer hrev -- bonefish mentionned that a clean fix will have to be very different, involve scripts instead of jamfiles

EDIT: found the build bot status at ​http://buildbot.haiku-os.org/ If ​this is the right log to look for then that'd be the error:

chmod: cannot access ‘/var/lib/buildbot/slaves/haiku/haiku-nightlies-x86_hybrid/generated/objects/haiku/x86/release/system/runtime_loader/runtime_loader’: No such file or directory

Not sure if it's a mismatch with/without a "_x86" suffix ? Or between the .so and the linked executable ?

[ttcoder]

ChMod ends up being executed before Ld ...

If reverting my suggested change and moving it right down above the last line instead,

+MODE on runtime_loader = $(EXEMODE) ;
+Chmod runtime_loader ;
+
 HaikuSubInclude arch $(TARGET_ARCH) ;

Then it builds:

/Haiku3/develop_hrev/haiku> jam -q runtime_loader
(..)
Ld /Haiku3/develop_hrev/haiku/generated/objects/haiku/x86_gcc2/release/system/runtime_loader/runtime_loader 
XRes1 /Haiku3/develop_hrev/haiku/generated/objects/haiku/x86_gcc2/release/system/runtime_loader/runtime_loader 
SetType1 /Haiku3/develop_hrev/haiku/generated/objects/haiku/x86_gcc2/release/system/runtime_loader/runtime_loader 
MimeSet1 /Haiku3/develop_hrev/haiku/generated/objects/haiku/x86_gcc2/release/system/runtime_loader/runtime_loader 
SetVersion1 /Haiku3/develop_hrev/haiku/generated/objects/haiku/x86_gcc2/release/system/runtime_loader/runtime_loader 
Chmod1 /Haiku3/develop_hrev/haiku/generated/objects/haiku/x86_gcc2/release/system/runtime_loader/runtime_loader 
...updated 1 target(s)...

This is not clean however (not guaranteed to work in parallel builds with jam -j I guess).. Looking into ChMod's internal, usage of Depends ..etc now ..

EDIT:

Seems I had an excessively paranoid view of "jam -j" :-)

It can run independant targets out of order, but it's not going to execute a given rule's lines out of order

It's clear from the ​original Jambase that rule Chmod and Link do not rely on any dependancy to execute in-order

And the OverridenJamRules's Link works the same

So Ld should be guaranteed to execute before Chmod if it's called before

[ttcoder]

So let's be done with this part of the discussion (but leave the ticket open for future completion of the other aspects) and apply the corrected two-liner:

diff --git a/src/system/runtime_loader/Jamfile b/src/system/runtime_loader/Jamfile
index 3a19cf1..e7b5251 100644
--- a/src/system/runtime_loader/Jamfile
+++ b/src/system/runtime_loader/Jamfile
@@ -106,4 +106,8 @@ Ld runtime_loader :
        : --no-undefined -shared -soname=runtime_loader
 ;
 
+MODE on runtime_loader = $(EXEMODE) ;
+Chmod runtime_loader ;
+
 HaikuSubInclude arch $(TARGET_ARCH) ;

EDIT 11 days later: I'm now building my own Haiku images and streaming them over a few days to Dane, restarting upload attempts until they succeed. Thanks for all the fish, I guess

[waddlesplash]

Applied in hrev51182, moving out of beta1 milestone.

[waddlesplash]

Confirmed that this was fixed for good by hrev51324/hrev51329. Finally. :)