Bash

[pittagurneyi]

Just some info, should someone not be aware:

find /home/sentrium/web/dev.packages.vyos.net/public_html/repositories/ -name *.deb -print

undergoes globbing by bash before the find command is executed, i.e. if there are *.deb files in your current working directory, then it will be expanded into them. Test with echo *.deb. Put "" or '' around it to solve the problem.

Should someone have the bash nullglob option active and no matching *.deb is found in the current working directory, then that becomes an empty string instead.

$ var="a b c"
$ printf '%s\n' $var
a
b
c

unless you put "" around the variable it is subject to all sorts of expansions and in this case word splitting, that bash does.

This becomes interesting when you use variables for paths and they contain whitespaces or other special characters that bash interpretes before executing your command. The kind of mess you can make if you are unlucky is huge.

$ path="/path/to something"
# Simulate what happens if you run  rm -rf $path:
$ for i in $path; do echo rm -rf "<$i>"; done
rm -rf </path/to>
rm -rf <something>

The same thing that would happen for the rm -rf command happens for the for bash command, i.e. the $path variable contents are split at the whitespace character and become two separate strings that are given as separate arguments for deletion to rm.

Same problem with cp, mv, etc.

There are exceptions to this rule, where bash takes care of it for you, but in general, just enclose any and all variables with "".

That also goes for nested variable usage:

echo "${array["$myindex"]}"
echo "${string%"$str"}"
. . .

If you want to write a wrapper script and it is given multiple arguments and not just one string, then make sure that you use this approach:

# Collect all arguments, properly escaped, in an array.
args=( "$@" )

# Compare entire string of all arguments:
#   Just compare:
#   printf '%s\n' "${args[@]}"
#   printf '%s\n' "${args[*]}"
#   The latter uses $IFS as separator for the arguments when stitching them together in a string,
#   i.e. that is what [*] does.
#   (IFS='.'; printf '%s\n' "${args[*]}")
if [[ "${args[*]}" == *"..."* ]]
then
...
fi

# Loop through the arguments:
for i in "${args[@]}"
do
...
done

# Append arguments:
args+=( your-argument1 your-argument2 ... )

# Build a new command by filtering out arguments:
mycmd=()
for i in "${args[@]}"
do
    [[ "$i" == "--something" ]] && continue
    mycmd+=( "$i" )
done

# Execute the command
"${args[@]}"

args=( echo a b c ); "${args[@]}" gives a b c as output. So no bash -c "$command" or something like it necessary. That just starts an unnecessary subprocess.

Use

#!/usr/bin/env bash

instead of #!/bin/bash to ensure the script can also run if bash isn't in /bin/bash, as is the case on some operating systems.

Furthermore, it is best practice to never use pure uppercase variables in your script, as they might conflict with environment variables.

This is a very good resource for getting an idea of when to look up the documentation:

• http://mywiki.wooledge.org/BashFAQ
• http://mywiki.wooledge.org/BashGuide

[dd010101]

I know the bash expansion can sometimes gets you but generally it's not issue, like command produces garbage output and you see it, you adjust it, I write commands to be fast for me to write thus that's also why I don't use proper shebang - it's just more to write and I know my system has bash.

I did update the examples so it's more proper as you say.

I'm not bash expert, I do use it for the simple stuff. I learn it on go when I find something I don't know. I'm programmer so I will never use bash for the complex things since I know I can write it in python much faster and the result will be much better.

So no bash -c "$command" or something like it necessary. That just starts an unnecessary subprocess.

Well if someone puts into $command pipes then the example would fail? You need at least eval for pipes? Here you see you need to think about what command can be, if you launch another bash you don't need to worry...

Furthermore, it is best practice to never use pure uppercase variables in your script, as they might conflict with environment variables.

That's actually a feature! If you use UPPERCASE, you can delete definition and you can define UPPERCASE in shell and your script will work without rewriting lowercase to UPPERCASE.

[pittagurneyi]

In your example of the uncron wrapper script you get the command, that is to be executed, via string. Then one way to do it would be, if no pipes or other forms of command sequences are part of the command:

# The variables are inserted by Jenkins before the command is ever given to ssh ...
arg1='reprepro -v -b ${VYOS_REPO_PATH} ${ARCH} includedeb ${RELEASE} ${SSH_DIR}/${FILENAME}'
read -ra args <<<"$arg1"
printf '%s\n' "${args[@]}"

Assuming you'd want to put safeguards in place before you blindly execute a given command, you can do that too. Then you can just use it as in my example.

Of course I fully expect the reprepro commands they used to be unsafe as well, as proper quotation is missing. In my opinion, that doesn't mean one can't take some care when building scripts that people are to run on their system - potentially not even inside a VM, as you'd mentioned.

Well if someone puts into $command pipes then the example would fail? You need at least eval for pipes? Here you see you need to think about what command can be, if you launch another bash you don't need to worry...

Correct, but they aren't using pipes as far as I can see and generally speaking I hope we don't encounter any such cases. Those are always bad.

That's actually a feature! If you use UPPERCASE, you can delete definition and you can define UPPERCASE in shell and your script will work without rewriting lowercase to UPPERCASE.

That was then just never how I'd resolved something. I always have lowercase variables in my script and set them to the environment variable should it be defined, if I want such functionality. The rest was, to me, always bad practice.

I know the bash expansion can sometimes gets you but generally it's not issue, like command produces garbage output and you see it, you adjust it, I write commands to be fast for me to write thus that's also why I don't use proper shebang - it's just more to write and I know my system has bash.

If I write commands that delete, move or copy stuff on the filesystem, I take care of doing it properly, or I don't do it at all. And you told me I was someone who just "wanted to get it done" (I don't remember the exact quote).

I'm not bash expert, I do use it for the simple stuff. I learn it on go when I find something I don't know. I'm programmer so I will never use bash for the complex things since I know I can write it in python much faster and the result will be much better.

"simple stuff", like potentially creating havoc on the filesystem and getting a faulty build if something doesn't work the way you'd expect it to?

Anyways ... message received. I will refrain from trying to correct the potentially more problematic code and keep my own scripts then.

If you have problems with me pointing out some of the risky code one can produce, were one to forget proper quoting, then writing an installation/automation script together makes no sense to me, as our prorities would never align, so I'll not be taking part in that.

I'll be closing this issue, forget I said anything.

[dd010101]

I did fix the proper bash shebang, and proper quotation for bash expansion, don't see the point about avoiding the bash -c - that seems like pedantic thing to me. But what you say does points to real issue that I don' have solution for.

The development version literally says that the build can compromise your OS so you need to use dedicated OS just for development. I do want to make a better version - it would be more complicated but also more proper.

That said - how would you solve this specific issue - where jenkins pulls unsafe code that can compromise your environment? The jenkins worker has full access to the shell where it runs and also it has full access to the SSH you give it - it can bypass uncron, bypass reprepro, it can execute anything by default and even if you did restrict it to reprepro that doesn't solve anything.

The main issue would be that such unsafe pull could generated compromised signed packages. It's easy for build script to poison your repository and you wouldn't even know. How would you improve on this? If the build can generate valid signed packages then it can always generated malicious signed packages as well and there is nothing you can do about that?

Replace malicious with broken, with accidental - it doesn't matter how does it happen - the point is that it can happen at all.

If I write commands that delete, move or copy stuff on the filesystem, I take care of doing it properly, or I don't do it at all. And you told me I was someone who just "wanted to get it done"

As you see the thing you suggest doesn't change anything at all in this matter. See above. I was referring to way you modified the build scripts, like return false in isCustomBuild(), that seems like JUST MAKE IT WORK to me :D.

ABSOLUTELY don't think I don't want your suggestions, I do, I just feel like you focusing on wrong issue here. You do have point, just the thing you focusing on doesn't solve it.

[pittagurneyi]

You are correct, that not much can be done at this time about the code that Jenkins produces, we'll have to - more or less blindly - trust VyOS and all the projects we pull data from.

Adding proper quotation to the Jenkinsfile's could be done, but that would mean more maintenance overhead.

By using a separate user on the build host for the build agent we reduce the risk of damaging the system. Other than that we can - hopefully - assume that the Jenkins developer put some form of safeguards in place as well.

That the user Jenkins runs as is non-root would help. Or as we are using it in a Docker container, we'll have safeguards in the form of cgroups in place, which Docker uses for the container separation.

The ssh access for placing the repository and iso files on the webhost can also be restricted by a separate user - I'm using the sentrium user for that. Furthermore one could, if one were so inclined, also build a proper wrapper for the ssh commands that are allowed to be executed, i.e. detecting malformed reprepro commands and forcing a users execution command, essentially giving all command-line arguments via ssh and ssh maps the users ssh-key/login to a host-side-specified script that then parses it. But that would mean more modifications for us.

As we'd only be using it ourselves, this particularly avenue makes little sense to me, I agree with you there. It also wasn't really the purpose of my post ...

All in all I just wanted to give anyone working on this project a very very very small idea of potential pitfalls and how to avoid them. The BashFAQ's list of pitfalls is far longer, though worth a read. Because I saw wrapper scripts recently that had problems, I just included a general example.

It was also not specifically directed at your wrapper script for uncron, that is a special case anyways, it was more for any automation/installation scripts we were thinking about writing and all build scripts for Jenkins, which are automatically executed, meaning normally we'd only look at the output, if it were to detect failures. However, if we write faulty code, then something might not work while throwing no errors, so we'll never see it until someone notices faulty functionality in VyOS.

So perhaps we have a misunderstanding. I only wanted to avoid problems that cause unnecessary headaches in all our files and commands.

I was referring to way you modified the build scripts, like return false in isCustomBuild(), that seems like JUST MAKE IT WORK to me :D.

What you got from me was more or less my first pass at fixing all the problems I had, to get everything working. It already took quite a lot of days, was done under lots of time pressure, because it lasted far longer than anticipated and I haven't had any time to revisit it, so this was the easy solution to see if it progressed to the next step. It has therefore just remaind there.

[pittagurneyi]

I thought by writing

If you want to write a wrapper script and it is given multiple arguments and not just one string, then make sure that you use this approach:

that I'd excluded your uncron wrapper ;) I could have been more specific in my word choice.

It was my mistake, that I had looked at it before and used bash -c "$command" as example of what was not necessary. I haven't looked again, that was in there, right?

[dd010101]

If it was meant just to look at all possibles bash use-cases and their pitfalls then sure, I didn't understand correctly. I will keep the FAQ around, seems like nice bash toolbox, read everything will take time...

Currently, for development purposes, everything runs as Jenkins user. It's not recommended but if you think about it then it actually doesn't matter. The only reason I can think of why Jenkins doesn't recommend built-it node is, that the pipeline has access to Jenkins itself. We don't care about that, because Jenkins doesn't have anything to access that the build doesn't access anyway... I will make improved instructions where agent, Jenkins and reprepro is separated into 3 users but that doesn't improve security for packages whatsoever. The only thing you are doing is to protect your OS or your Jenkins and if you share OS and share Jenkins with for other projects then that would protect the other projects from bad build but it doesn't do anything for vyos packages.

Thus if you have VM only for vyos stuff then you may run everything in one VM as root and it's as save as having 3 VMs and limited users, because in both cases bad build can mess with packages and that's basically only thing what matters. You may think that if build manages to delete/corrupt your Jenkins or your OS that's bad but actually that's okay - because then you know something wrong happened and it's easy to fix (if you do backups!). If you goal was to have high uptime/availability then you may care if your system gets disabled and then it would make sense to protect the OS/Jenkins. But in this situation if bad build silently breaks or infects packages - then you don't even know, that's the real problem and not possible to solve.

Thus the best one can do is to secure it's own GIT repositories that Jenkins pulls from as possible and hope the upstream repositories do it as well... Bugs are impossible to avoid so bad code that gets pulled in terms it's just broken is something what we need to live with.

Do you remember all these magic sequences?

parameter     result
-----------   ------------------------------
$name         polish.ostrich.racing.champion
${name#*.}           ostrich.racing.champion
${name##*.}                         champion
${name%%.*}   polish
${name%.*}    polish.ostrich.racing

parameter     result
-----------   --------------------------------------------------------
$file         /usr/share/java-1.4.2-sun/demo/applets/Clock/Clock.class
${file#*/}     usr/share/java-1.4.2-sun/demo/applets/Clock/Clock.class
${file##*/}                                                Clock.class
${file%%/*}
${file%/*}    /usr/share/java-1.4.2-sun/demo/applets/Clock

parameter     result
-----------   --------------------------------------------------------
${string^}    Hello, World! # First character to uppercase
${string^^}   HELLO, WORLD! # All characters to uppercase
${string,}    hello, World! # First character to lowercase
${string,,}   hello, world! # All characters to lowercase

## %% ^^ ,, Not very intuitive.

I was well aware of "$1" vs $1, these things could do some nasty result in specific situations but generally it will just make the script not work, I didn't ever manage to rm -rf anything by mistake via script. I did make typo when back when and in shell run rm -rf / some/thing, such lesson did teach me to be careful with rm... Shell without /bin:/usr/bin isn't fun I do remember.

[pittagurneyi]

Do you remember all these magic sequences?

Once you've used them often enough, it's quite easy. They are the fastest way to achieve your goal.

The problem with bash is that every call to an outside function, i.e. sed, awk, etc. adds such overhead when called multiple times, that I really try to avoid it.

Don't get me wrong, in some ways I really do hate bash. So many edge cases, problematic things that can and will go wrong unless you are very careful in all but very simple scripts ... Also it is slow as f*ck.

But, I don't have to do maintenance on them, like ever ... No one will ever dare break bash syntax. Every system administrator on the planet would be so angry.

That is the only reason I write mostly bash for all system administration tasks. Because I don't have the time to always migrate python, etc. scripts if something in the syntax, modules, etc. changes again.

Now I know that the python basics don't really change that much, but it is the fact, that they actually might change and if I depend on the scripts, then it does matter to me. Knowing the most fundamental scripts that take care of maintenance, upgrades, backups, etc. aren't something I have to worry about between system upgrades, is a huge weight lifted from my shoulders.

I saw that you wrote me something regarding the bash script(s) you created. I'll gladly help and take a closer look, but I have so many other things on my plate right now, that it will take a week or two until I can get to it.

[dd010101]

I did make some scripts to manage Jenkins in bash just to try it, you can check those in repo if you are bored in the future, but working with JSON (jq) or XML (xmlstarlet) is indeed slow and unfriendly, because you are forced to call extra processes just to handle piece of JSON.

This depends on what tasks you do... I do a lot of programming so I do like bash for the simple things but for more complex tasks like these I normally use Python. I used many scripting/programming languages and I end-up with like 4 languages I use regularly depending on what I want to do where. I see the selection of language like selection of tool - you wouldn't use hammer on screw and that's how I feel when I see hundreds lines of bash in one place...

Python is good fit for scripting since it has built-in libraries that can do lot without additional packages, also Linux distributions use Python themselves, so it's not like you need to install it - it's already there - use it! Python is also very stable - I don't even remember many BC breaks, the 2to3 move was break but it wasn't hard to migrate anyway...

On the other hand - I do willingly use other language daily that breaks your code literally with every new version - why would you do that? I can do the same tasks in multiple different languages but this one language can do specific task so efficiently it saves you so much time, you can afford the overhead and still be ahead. At the end it doesn't matter how you do it - but if some specific language can do more with less development time and less errors then it just make sense to use it with whatever disadvantages it has. This of course doesn't have much to do with the language itself - like the syntax, the language is as good as the built-in and 3rd party libraries are, that's what makes the platform efficient for given task - not the syntax.

[pittagurneyi]

I know quite a lot of programming languages, that isn't my problem. I've used python for a specific problem just last week. Wouldn't have ever tried solving it in bash, that would have been a disaster. Especially considering all the nice modules that other people developed and maintain, which help you solve so many different problems without much work.

Of course you always use the most appropriate tool for the job. Yes, python didn't have many breaks, I've been using it for many many years myself. The problem I was describing was simply: If you are the only one maintaining a certain set of scripts with in total perhaps 20.000 lines of code, and they perform maintenance tasks for you more or less 24/7 or you use them weekly and you depend on them working flawlessly, knowing you can not guarantee you will always have time to check them all before an emergency system upgrade must be performed due to some zero day vulnerability in the kernel, a system library, etc. then your priorities shift quite severly.

For all other tasks I have no problem with evolving programming languages, as long as I don't require it to function always, after every system upgrade, no matter the time of day.

[dd010101]

From my point of view something you describe is statistically impossible - there is no way to write code without errors, even if everything you depend on is perfect, humans do mistakes and the things you depending on aren't without their issues, because humans were involved there as well. Thus I don't expect that everything will work - my expectation is - everything can fail, thus I need to have robust error handling and reporting, so the system can recover and/or call for help, because I can't make sure nothing ever fails, I know something will fail eventually... Why you think the bash, all utilities and all those 20k lines are without issues forever? The idea you make system upgrade and you expect it to always work... I don't think that's a option you have...

[pittagurneyi]

I don't. I just don't want them to unexpectedly stop functioning due to a system upgrade. If they misfunction due to programming errors, which can always happen, as I'm human, then I'll deal with it. But if they've been working without major issues for many years and you don't have the time to fix problems introduced by changes in the programming language/its modules, then your options shrink, wouldn't you agree?

The last time I used one of the python modules I needed last week, was a year ago. They changed the syntax of certain function calls so much in the last year that I had to fix at least 4 or 5 such issues before it began working again. I don't have time for such idiocy. Something is working, don't touch it for gods sake or do it in such a way that peoples script don't break by keeping backwards compatible syntax. But no, most things need to evolve and new functionality has to be added. There is a place for innovation, that is true, but there also has to be something you can depend on continuing to function as it always has been. Honestly, the only thing I've found is bash, which just comes with very basic functionality. But at least that stays the same.

That is why I use bash for all my basic server functionality, because I, working along, don't have the time or nerve to deal with it constantly.

Also, yes you can indeed do error handling in bash and at least guarantee that if it fails you'll be notified. Of course you don't have nice things like error handlers, or such things, but basic functionality is there.

[dd010101]

I think you aren't fair to the snake. Python doesn't change API every year that's ridiculous, in fact the built in libraries nearly never change. If your experience is such then it's skewed by bad luck. That's not the norm. Many scripts in Linux distros are written in python as well as in vyos and do you see them releasing new packages because of python - I don't.

I run many scripts like cron or services for ages without issues related to python or libraries. Basically the thing what brakes python is when you change python version, for me that's when Debian releases new version - not that often. In those cases I need to rebuild venv and very rarely I need to change code, sometimes I update librsries - those are wild cards, sometimes they change API other times not, it depends... If you would use naked python then it would work without changes tho. So you can make python code that will survive for very long time if you don't use pip/system libraries and only use built in ones.

Is bash more future proof, yes but it depends - if I can make python script more robust because you know, you have proper error handling - not just exit codes, and I do it in less time. I may save enough time for ages of updates... Do python scripts need more babysitting in 10 year lifespan, for sure. Do you spend more time overall - that depends, I for sure save time with python even if I need to babysit every couple years a little.

[pittagurneyi]

I don't say that base python changes API every year. If you read carefully, I said that a python module did that. I also never said it was a core module.

You say you use debian, I have my servers running a rolling release, so I get a new python version basically every year. And no, I won't go into detail on why or discuss it with you.

Do you spend more time overall - that depends, I for sure save time with python even if I need to babysit every couple years a little.

You don't seem to get it. I don't give a f*ck about time spent on it. I just need it to never break with system upgrades. If I have to upgrade 20 servers in one evening/night and get unforseen consequences because something broke somewhere, I have a huge problem. That can not happen. I'm also now done discussing it.

I never said anywhere, that you can't get the same thing done with python faster. There are always tradeoffs. Just choose your priorities and select accordingly. Done.

[dd010101]

You have conflicting needs - running unstable and requiring stable. Okay if you want to build stable script on unstable distribution then you backed yourself into corner where basically nothing else works.

How anyone can think that frequent updating is way for stable system, that's just beyond me. Changing means breaking, that's how it works.

I do have issues when upgrading due to random config changes of some random services and such. Why would I make myself to do it more often? Because I want to save time of course!

[pittagurneyi]

Okay if you want to build stable script on unstable distribution then you backed yourself into corner where basically nothing else works.

Where did I ever say I'm running an unstable distribution, just what kind of idiot do you think I am? I'm not using ArchLinux or something like it on my production servers. Also I have my reasons. Again, you want to have a discussion with me about something I don't want to discuss and are also again assuming things that aren't true. You aren't me, you don't know my systems, my requirements and my reasons. I didn't give you details. Stop arguing with me for no reason other than to argue. And please start reading my sentences more carefully. I take time in constructing my answers so as to avoid misunderstandings and confusion. You seem to strive off it. I want to stop the discussion because it is unproductive and we aren't really disagreeing about anything. You try to start it up again by misreading what I wrote, again.

I won't be responding to you again. I have better things to do. Farewell for now.

[dd010101]

There is issue with communication.

You say module, like everything in python is module, literally everything including the built in classes. I don't read? You don't say anything specific.

You don't say you need to have distribution that changes versions all the time. Did you said? No.

How I can read something you don't say?

Why would I even expect you running Arch or Gentoo because you want stable system? You didn't say so...

So you don't say anything specific thus I need to guess and then you don't like if I assume the normal things like for stable system you would run something that has LTS...

Sure I didn't read something you didn't communicate. As you say you don't want to say anything specific so you don't and you force people to assume common things about your not so common use case.

If you said look I have this very specific use case that you will never see in the wild, it's not typical whatsoever then perhaps I could understand but you didn't say that did you?

I can read as much as I can but I can't read something you don't say.

Don't reply just take away that if you force people to assume don't be surprised they can't read your mind and they will assume the most common thing for them.