Category: Linux
Linux find out what a process is doing
- start top
- lsof -p <pid>
- strace -p <pid>
- ltrace -p <pid>
Happy debugging 😀
Zenoss reverse proxy with Pound (CentOS)
Zenoss don’t support SSL certificates out-of-the-box. If you want to use an SSL connection to your zenoss monitor server the only thing you can do is use an reverse proxy. You can use this howto to install and configure a pound reverse proxy.
Install pound with the EPEL
Install the EPEL (more info about EPEL) repository with these commands:
su -c 'rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm'
yum update
Install pound
yum install pound
Install pound without the EPEL
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/Pound-2.6-2.el6.x86_64.rpm rpm -ivh Pound-2.6-2.el6.x86_64.rpm
Configure Pound
I had a lot of trouble because I used a real SSL certificate immediately. The cause was I dropped the SSL cert in the wrong linux folder. Best practice is first create a selfsigned SSL, test pound and then replace the selfsigned with a real SSL certificate.
cd /etc/ssl && openssl req -x509 -newkey rsa:1024 -keyout local.server.pem -out local.server.pem -days 365 -nodes
Configure Pound
nano /etc/pound.cfg
Config file:
User "pound" Group "pound" Control "/var/lib/pound/pound.cfg"
ListenHTTPS Address 192.168.0.x Port 443 Cert "/etc/ssl/local.server.pem" End
Service BackEnd Address 127.0.0.1 Port 8080 End End
Now start the pound service
service pound start
Change the Zenoss config the handle the HTTPS traffic
nano /opt/zenoss/etc/zope.conf
Ad these 3 lines:
<cgi-environment> HTTPS ON </cgi-environment>
Restart zope
su - zenoss restart zopectl
Replace the selfsigned SSL with a wildcard SSL (optional)
Create a PFX in windows. Tranfer the PFX to the Zenoss server and tranform the PFX to PEM (Linux certificate format). The command:
openssl pkcs12 -in validcertificate.pfx -out wilcard.domain.nl.pem -nodes
Now change the pound cert:
nano /etc/pound.cfg
ListenHTTPS Address 192.168.0.x Port 443 Cert "/etc/ssl/wilcard.domain.nl.pem" End
Restart the service
service pound restart
Source: Enabling SSL in Zenoss 4.2 – Open Source Network Monitoring and Systems Management
Zenoss: Performance issues with too many events
When you have too many events in your zenoss environment the zenoss webinterface will be very sloooooooow. And you get all kind of errors:
- Script don’t respond
- Connection refused. Check zeneventserver status on deamons
- A zenoss error has occurred
When you start top you see alot of java executables when you click on the Infrastructure zenoss button. Java sometimes take 350% CPU.
java -Djetty.host=localhost -server -XX:+HeapDumpOnOutOfMemoryError -DZENOSS_COMMAND=zeneventserver -DZENHOME=/opt/zenoss -Djetty.home=/opt/zenoss -Djetty.logs=/opt/zenoss/log -Dlogback.configurationFile=/opt/zenoss/etc/zeneventserver/logback.xml -Xmx1024m -DZENOSS_DAEMON=y -jar /opt/zenoss/lib/jetty-start-7.5.3.v20111011.jar --config=/opt/zenoss/etc/zeneventserver/jetty/start.config --ini=/opt/zenoss/etc/zeneventserver/jetty/jetty.ini --pre=etc/zeneventserver/jetty/jetty-logging.xml
I’ve read a lot of zenoss documentation on the internet but didn’t found a nice article to get rid of all the events. So here is an article how I fixed it.
Basic steps:
- Backup Zenoss
- Stop zenoss
- Create a new zeneventserver database
- Remove zeneventserver content
- Restore the zenoss backup
- Start zenoss
- Enjoy your fast zenoss 🙂
Detailed steps:
- ssh zenoss host
- Switch zenoss user
# su zenoss
- Create backup:
$ /opt/zenoss/bin/zenbackup -v10
- Stop Zenoss service
$ zenoss stop
- edit the zeneventserver script
nano /opt/zenoss/bin/zeneventserver-create-db
- Search for root and add the root password
- Run the script
$ zeneventserver-create-db --force --dbtype=mysql
- Clear the zeneventserver folder
rm -rf $ZENHOME/var/zeneventserver/*
Now some tricky part. Zenoss change some MySQL passwords when you do a restore. This result in a access denied for user [email protected] during a restore. There is a fix for this problem. Reset the [email protected]’localhost’ and [email protected]’%’ MySQL passwords before you do a restore.
First get the current mysql from the global.conf file (yellow). This password is the password you need for the restore.
$nano /opt/zenoss/etc/global.conf
Tip: Too check the password (encrypted). You can do the same after you change the password:
$ mysql -uroot -p
mysql> select * from mysql.user;
Now reset the password
SET PASSWORD FOR 'zenoss'@'localhost' = PASSWORD('BEagPxxxxxxxxxxxxxxx');
SET PASSWORD FOR 'zenoss'@'%' = PASSWORD('BEagPxxxxxxxxxxxxxxx');
When you check the permissions now you see another encryption:
$ mysql -uroot -p
mysql> select * from mysql.user;
(I don’t have an image example because this is an production enviroment)
Optional: To check the zenoss user permissions:
mysql> SELECT user, host, db, select_priv, insert_priv, grant_priv FROM mysql.db;
Optional: When you still have errors or the above rights ain’t good try these two MySQL scripts:
mysql> CREATE USER 'zenoss'@'%' IDENTIFIED BY 'some_pass';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'zenoss'@'localhost'
-> WITH GRANT OPTION;
mysql> CREATE USER 'zenoss'@'%' IDENTIFIED BY 'some_pass';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'zenoss'@'localhost'
-> WITH GRANT OPTION;
Ok, now everting is set do a restore. The -v stands for verbose and with the no-eventsdb you don’t restore all the events. That’s exactly what we want
zenrestore --file=/opt/zenoss/backups/zenbackup_2014013 -v --no-eventsdb
Now start zenoss
$zenoss start
That’s it. Enjoy the performance and set some parameters that your events ain’t that big any more in the future.
Source: http://www.nickyeates.com/technology/zenoss/useful_commands_zenoss4
Source: http://community.zenoss.org/docs/DOC-3048
IPtables howto
I copy/paste this howto from the debian wiki @ https://wiki.debian.org/iptables and add this to my blog to for fast finding the best iptables article on the net.
Iptables provides packet filtering, network address translation (NAT) and other packet mangling.
Two of the most common uses of iptables is to provide firewall support and NAT.
Configuring iptables manually is challenging for the uninitiated. Fortunately, there are many configuration tools (wizards) available to assist: e.g., fwbuilder, bastille, ferm (wiki page), ufw (Uncomplicated Firewall, from Ubuntu).
Viewing current configuration
See what rules are already configured. Issue this command:
iptables -L
The output will be similar to this:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
This allows anyone access to anything from anywhere.
Storing iptables rules in a file
Note: there is a package designed to help with this: iptables-persistent
Let’s tighten that up a bit by creating a test iptables file:
nano /etc/iptables.test.rules
In this file enter some basic rules:
*filter # Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0 -A INPUT -i lo -j ACCEPT -A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT # Accepts all established inbound connections -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # Allows all outbound traffic # You could modify this to only allow certain traffic -A OUTPUT -j ACCEPT # Allows HTTP and HTTPS connections from anywhere (the normal ports for websites) -A INPUT -p tcp --dport 80 -j ACCEPT -A INPUT -p tcp --dport 443 -j ACCEPT # Allows SSH connections # THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE -A INPUT -p tcp -m state --state NEW --dport 30000 -j ACCEPT # Now you should read up on iptables rules and consider whether ssh access # for everyone is really desired. Most likely you will only allow access from certain IPs. # Allow ping -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT # log iptables denied calls (access via 'dmesg' command) -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7 # Reject all other inbound - default deny unless explicitly allowed policy: -A INPUT -j REJECT -A FORWARD -j REJECT COMMIT
That may look complicated, but look at each section at a time. You will see that it simply shuts all ports except the ones we have allowed – which in this case are ports 80 and 443 (the standard web browser ports) and the SSH port defined earlier.
Activate these new rules:
iptables-restore < /etc/iptables.test.rules
And see the difference:
iptables -L
Now the output tells us that only the ports defined above are open. All the others are closed.
Once you are happy, save the new rules to the master iptables file:
iptables-save > /etc/iptables.up.rules
To make sure the iptables rules are started on a reboot we’ll create a new file:
nano /etc/network/if-pre-up.d/iptables
or in the the if-up.d if the if-pre-up.d not work
nano /etc/network/if-up.d/iptables
Add these lines to it:
#!/bin/bash /sbin/iptables-restore < /etc/iptables.up.rules
The file needs to be executable so change the permissions:
chmod +x /etc/network/if-pre-up.d/iptables
Show spam script on linux webserver
Sometimes a website on a shared server is compromised by a hacker. Most of the time they use the website to spam the WWW around. It can be hard to find the specific script who is responsible for the spam. Here I will describe a method you can use to find and eliminate the script.
First stop postfix to stop the mail flood
# service stop postfix
Now get the mailque and write down the message id (yellow)
# mailq
Get the message header
# postcat -vq A9CC9182699 |more
Now see the yellow text above. You have find the culprit. Now find the file location:
# find /var/www/web/ -name *addwp.php*
Eliminate the process and patch the website.
Delete the queue (USE THIS WITH CAUTION!!!!)
# postsuper -d ALL
And start postfix
# service start postfix
To find and kill a spamming perl script try this:
Find the process:
lsof -i :25 readlink -f /proc/{PID of process}/exe kill PID
Check and delete the (sendmail) QUEUE:
cd /var/spool/mqueue/ ls nano queue file rm -f *
It can be very difficult to find a specific perl spam script because the path is not always visible. To find the script do a locate and isolate/rename the files.
locate *.pl |grep /var/www/
For more commands see my blog article: https://blog.wapnet.nl/2014/04/linux-find-out-what-a-process-is-doing/
I found another nice tool to find suspicious scripts: http://cbl.abuseat.org/findbot.pl
Zenoss Backups
Create a zenoss user cron
sudo crontab -u zenoss -e
Use this lines to backup every day 10 AM.
HISTFILE=/home/zenoss/.bash_history
HOME=/home/zenoss
INSTANCE_HOME=/opt/zenoss
LD_LIBRARY_PATH=/opt/zenoss/lib
LOGNAME=zenoss
MAIL=/var/spool/mail/zenoss
PATH=/opt/zenoss/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/zenoss/bin
PWD=/home/zenoss
PYTHONPATH=/opt/zenoss/lib/python
USER=zenoss
ZENHOME=/opt/zenoss
30 10 * * * /opt/zenoss/bin/zenbackup -v10
Note: When I not put the variables to the cron I get this error:
ERROR: $ZENHOME is not set.
This is usually caused by executing this command as root rather than as the zenoss user. Either define $ZENHOME or run this command as a different user.
Because I can’t find a nice solution on the internet I fix it to add the SET lines to the cron.
Create a cleanup job
sudo crontab -e
Use this line to cleanup the backups every sunday 12 AM
00 12 * * 7 root /usr/bin/find /opt/zenoss/backups -mtime +30 -type f -exec rm \{\} \;
Crunch – wordlist / password generator for brute force
Crunch is a wordlist generator where you can specify a standard character set or a character set you specify. crunch can generate all possible combinations and permutations.
Features
- crunch generates wordlists in both combination and permutation ways
- it can breakup output by number of lines or file size
- now has resume support
- pattern now supports number and symbols
- pattern now supports upper and lower case characters separately
- adds a status report when generating multiple files
- new -l option for literal support of @,%^
- new -d option to limit duplicate characters see man file for details
- now has unicode support
See the sourceforge project: crunch – wordlist generator
Linux clear disk cache from RAM
Sometimes your memory is full and youu can find the reason. 99/100 times the cause is that all your RAM is used for disk cache. This is an standard linux feature. With this command you can free your disk cache.
echo 3 | sudo tee /proc/sys/vm/drop_caches