Category Archives: Uncategorized

Nginx Virtual Host and Reverse Proxy

Firstly, there’s no such thing as Virtual Host in nginx. , Virtual Host is an apache terminology.

Scenario:

  • Domain mycooldomain1.com pointing to VPS server
  • Nginx running on port 80
  • Tomcat running on port 8080
  • Only inbound TCP traffic to port 80 is allowed through firewall

In your nginx.conf (mine’s on /etc/nginx/nginx.conf), add following inside the http element:

http {
  ...
  server {

    server_name mycooldomain1.com;
    access_log /var/log/nginx/mycooldomain1.com.access.log main;
    error_log /var/log/nginx/mycooldomain1.com.error.log;

      location / {
        proxy_pass http://localhost:8080;
        proxy_redirect default;
        proxy_cookie_domain localhost mycooldomain1.com;
      }
  }
  ...
}

The server_name and location / expression matches request to http://mycooldomain.com while proxy_pass sets the backend where the response will be fetched from.

proxy_redirect ensures any 3xx redirects and Location: header on response is rewritten into mycooldomain1.com.

If your backend has different context root (eg: http://mycooldomain.com to http://localhost:8080/someapp) you will also need to adjust the cookie path

proxy_cookie_path /someapp/ /;

DOS Script To Cap Log Files

I have this common problem where my servers generates about 5-6 megs of log file every day filling up the disk, and following dos script to delete old logs have been quite handy

:: Iterate files alphabetically at specified folder and keep a maximum of N to 
:: avoid filling disk space. Called by run.bat
:: This script takes 3 arguments
 off
setlocal ENABLEDELAYEDEXPANSION

set FOLDER=%1
set FILEPREFIX=%2
set LIMIT=%3

echo Accessing %FOLDER%
cd %FOLDER%

set /a COUNT=0
for /f "tokens=*" %%f in ('dir /b /a-d-h-s /o-n %FILEPREFIX%*') do (
          set /a COUNT=COUNT+1
          if !COUNT! GTR %LIMIT% (
                        echo deleting %%f
                        del /f /q %%f   
                )
        )
endlocal

I place the script above in C:filecleanerdeletefiles.bat. This script iterates a folder alphabetically and keep only specified amount of files. I then created a second script to be called by task scheduler

:: Called by task scheduler. This script calls deletefiles.bat over each file path prefix
::
:: Example:
:: deletefiles.bat c:logrotate-test access.log 3 
:: means keep maximum 3 files (sorted alphabetically) starting with access.log
 off

deletefiles.bat "C:apache-tomcat-6.0.35" catalina 7
deletefiles.bat "C:apache-tomcat-6.0.35" commons-daemon 7
deletefiles.bat "C:apache-tomcat-6.0.35" host-manager 7
deletefiles.bat "C:apache-tomcat-6.0.35" localhost 7
deletefiles.bat "C:apache-tomcat-6.0.35" manager 7
deletefiles.bat "C:apache-tomcat-6.0.35" tomcat6-stderr 7
deletefiles.bat "C:apache-tomcat-6.0.35" tomcat6-stdout 7

I put this script in C:filecleanerrun.bat. As you can see it calls deletefiles.bat several times to clean my tomcat log files. The script uses following arguments:

  1. The folder where the log files exist
  2. The prefix of the log files
  3. How many latest files should be kept

It’s important to note this will only work if the files alphabetical ordering implies their age. This typically works best if the files pattern has a yyyy-MM-dd format (or similar at the end):

stdout.2013-12-04.log
stdout.2013-12-05.log
stdout.2013-12-06.log
stdout.2013-12-07.log

Finally to run this automatically every 3.30am in the morning I created a task scheduler with following action:

filecleaner

Using File Protocol to Deploy Maven Project To Windows Shared Folder

Our development team is fairly small and we’re not at the point where we need Nexus yet, so I decided we can try using a simple Windows server share as our internal Maven repository.

On each of our shared Maven project pom.xml we add following distributionManagement configuration:


  
    enfinium
    Enfinium Internal Repository
    file://mavenrepo/maven_repository
  

Where //mavenrepo/maven_repository is the Windows server share we’ve setup. We’ve made sure each team member has correct read/write permission.

However every time we deploy Maven would say everything is successful but the file is nowhere to be found on the remote repository. (We’re using Maven 3.1.0

Turns out with file protocol and Windows server share, this is the syntax that works for us (yes Maven just fail silently and said everything was SUCCESSFUL)

file:////mavenrepo/maven_repository

Multithreaded Server in Java

In Java, listening over TCP can be simply done using ServerSocket. Example below binds our program to port 5656:

ServerSocket serverSocket = new ServerSocket(5656);

To access the client’s input stream, we first need to accept the connection and obtain the Socket object. This is done through the accept method:

ServerSocket serverSocket = new ServerSocket(5656);
Socket socket = serverSocket.accept();

The program above can only accept 1 client connection. We want to able to accept infinitely many connections, hence we can use an infinite loop to keep accepting new connections:

ServerSocket serverSocket = new ServerSocket(5656);
while(true) {
  Socket socket = serverSocket.accept();
  // process client connection..
}

However once a client is connected our code has to wait until he/she disconnects before we can accept the next one. This might be ok if the work duration is very short, but if it’s very long (eg: a chatting program) then this won’t do. To solve this we can use an ExecutorService which schedules work to be done in the future (in a separate thread) and allowing the calling thread to progress. In our case we will use a fixed thread-pool executor service, which effectively limit how many active connections we can have. We will soon write the ConnectionHandler class below to process the connection.

ExecutorService executorService = Executors.newFixedThreadPool(100);
ServerSocket serverSocket = new ServerSocket(5656);
while(true) {
  Socket socket = serverSocket.accept();
  executorService.execute(new ConnectionHandler(socket));
  // go back to start of infinite loop and listen for next incoming connection
}

The ConnectionHandler class above implements Runnable and does the work of listening & writing to client. Here’s an example of what simple ConnectionHandler that just echoes messages sent by client back:

public class ConnectionHandler implements Runnable {

  private Socket socket;

  public ConnectionHandler(Socket socket) {
    this.socket = socket;
  }

  public void run() {
    BufferedReader reader = null;
    PrintWriter writer = null;
    try {
      reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
      writer = new PrintWriter(socket.getOutputStream(), true);

      // The read loop. Code only exits this loop if connection is lost / client disconnects
      while(true) {
        String line = reader.readLine();
        if(line == null) break;
        writer.println("Echo: " + line);
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    } finally {
      try {
        if(reader != null) reader.close();
        if(writer != null) writer.close();
      } catch (IOException e) {
        throw new RuntimeException(e);
      }
    }
  }

}

Keep in mind the run method will run in a separate thread, not the main thread, so make sure you synchronize reads / writes into shared resources.

Socket Timeout

Ok so far so good, we can handle multiple connection simultaneously in separate threads. But our code is still vulnerable. Recall our thread pool setting is capped at 100 which means we can only accept 100 simultaneous connection. This leave us vulnerable to denial-of-service attack, a hacker could easily open 100 connections and leave it be — denying access to other clients. One way to mitigate this is to set socket timeout:

ExecutorService executorService = Executors.newFixedThreadPool(100);
ServerSocket serverSocket = new ServerSocket(5656);
while(true) {
  Socket socket = serverSocket.accept();
  socket.setSoTimeout(3000); // inputstream's read times out if no data came after 3 seconds
  executorService.execute(new ConnectionHandler(socket));
  // go back to start of infinite loop and listen for next incoming connection
}

By setting the socket timeout, the BufferedReader.readLine() method used by ConnectionHandler above will throw SocketTimeoutException if it hasn’t had any data in 3 seconds.

public void run() {
  BufferedReader reader = null;
  PrintWriter writer = null;
  try {
    reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
    writer = new PrintWriter(socket.getOutputStream(), true);
 
    while(true) {
      // SocketTimeoutException thrown here if nothing read after 3 seconds
      String line = reader.readLine();
      if(line == null) break;
      writer.println("Echo: " + line);
    }
  } catch (SocketTimeoutException e) {
    System.out.println("Connection timed out");
  } catch (IOException e) {
    throw new RuntimeException(e);
  } finally {
    try {
      if(reader != null) reader.close();
      if(writer != null) writer.close();
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }
}

mount.nfs: Input/output error When Mounting From CentOS 5.9 to 6.4

Was trying to mount a NFS shared folder on CentOS 6.4 from CentOS 5.9 and found a really tricky problem. I’ve added the /etc/fstab entry, but once I ran mount /my/folder it stuck for 2-3 minutes and came back saying

mount.nfs: Input/output error

Things I’ve Tried

This is all the things I’ve tried with no success:

  • Ensured network connection was ok. I checked the firewall / iptables and made sure the client machine can connect to NFS server
  • Checked my /etc/hosts file, ensured no dodgy entries
  • Double checked /etc/exports and ran exportfs -r on the NFS server
  • Ran showmount -e [host_ip] to check the NFS server really does advertise the shared folders
  • Ran rpcinfo -p [host_ip] to check the version and supporting services are available
  • Rebooting the server many times

Solution

Thanks to , the solution was to add nolock option to the /etc/fstab file.

10.0.10.10:/my/folder /mnt/nfs/my/folder nfs nolock 0 0

CentOS manual is pretty vague as well on what this option really does, but oh well it seems to do the job for time being

nolock — Disables file locking. This setting is occasionally required when connecting to older NFS servers.

CentOS / RHEL NFS Share Network Folder Mounting

Scenario

We are building a second web server with a hope of installing a load balancer soon. We came into a problem where we have to synchronize the file contents between multiple servers. The solution we’re opting is to install and mount a shared network folder (NFS) visible from both web servers. The shared folder lives on its own server (aka file system server).

It is assumed:

  1. The file system server host (Master) is 12.0.10.10
  2. The client is 12.0.20.20
  3. The folder to be shared on master is /var/www/vhosts
  4. The folder will be mounted to /mnt/nfs/var/www/vhosts on client

Creating a NFS Share on the File System Server

Perform these steps on the file system server (aka “Master”):

  1. Install nfs-utils and nfs-utils-lib package using yum:
    yum install nfs-utils nfs-utils-lib
    Use yum list installed to check if the packages are already installed
  2. Turn on nfs service so it automatically starts on boot
    chkconfig nfs on
    Use chkconfig --list to check status of all services
  3. Start rpcbind and nfs service
    service rpcbind start
    service nfs start
  4. Edit /etc/exports file and add following line at the end
    /var/www/vhosts 12.0.20.20(rw,sync,no_root_squash,no_subtree_check)
    See here for explanation of the options
  5. Run following command to apply the changes
    exportfs -a

Mounting it on the Client

Following steps will mount the network folder permanently (folder will automatically re-mount on server reboot). Perform this on the client server:

  1. Similar to server, ensure nfs-utils and nfs-utils-lib are installed
    yum install nfs-utils nfs-utils-lib
  2. Create the directory that will hold the mount (mount point)
    mkdir -p /mnt/nfs/var/www/vhosts
  3. Edit /etc/fstab file and append following line
    12.0.10.10:/var/www/vhosts /mnt/nfs/var/www/vhosts nfs defaults 0 0
  4. Run mount /mnt/nfs/var/www/vhosts to apply the mounting. Check if the mounting is successful using df -h

OS User on Master and Client

To ensure file system operations are consistent, consider propagating same OS user setup between the master and client. Note that although same username exists on both master and client they don’t necessarily have the same UID.

  • Checking uid and group of user jim:
    $ id jim
    uid=506(jim) gid=505(xyzco) groups=505(xyzco)

    (jim has uid 506 and belongs to group xyzco)
  • Adding a new user jim with uid 506
    useradd -u 506 jim
  • Adding jim to the group xyzco
    usermod -G xyzco jim
  • Setting / resetting password for jim
    passwd jim

Thanks To

Additional Reading

Managing Services on UNIX Using chkconfig

A typical unix service-related command looks like this

# start mysql service
service mysqld start

# stop mysql service
service mysqld stop

The list of currently configures service can be viewed using chkconfig command

[root ~]# chkconfig --list
...
ip6tables       0:off   1:off   2:on    3:on    4:on    5:on    6:off
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
messagebus      0:off   1:off   2:on    3:on    4:on    5:on    6:off
mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
...

Each of the number 0-6 corresponds to different OS runlevel.

To disable/enable mysqld service, use following command

# disable mysqld
chkconfig mysqld off

# enable mysqld
chkconfig mysqld on

By default chkconfig on/off will set the service into 2-5 runlevel.

Backing Up and Restoring MySQL Database Using mysqldump

Although not always the most efficient, mysqldump is a handy way to migrate your MySQL server — particularly if you don’t have many schemas and the size of data is small.

To create backup, first list all schemas on the server using

SHOW DATABASES;

I normally avoid backing up performance_schema, information_schema and mysql schema. They contain database configuration, user settings etc. This would mean you have to reconfigure all your settings on the new server

Then take the dump of all schema using this command:

mysqldump -u***** -p***** --databases  > mybackup_20131107_1223.sql

Then compress, transfer and inflate the sql file into the new server’s host. When you’re ready to restore, just do

mysql -u***** -p***** < mybackup_20131107_1223.sql

Windows 7 Unable to Import Trusted Root Certification Authorities

Windows has its own trusted root CA certificates (which is also used by Google Chrome browsers). If your company is running your own internal CA, an additional CA certificate has to be added to it.

I encountered a very strange problem where everytime I import it via certmgr.msc the dialog says The import was successful but the certificate was never imported.

Solution 1: Use Administrator Elevated Privilege

  1. Click on windows icon -> Search for cmd
  2. Right click cmd.exe -> Run as Administrator
  3. From the command prompt, run certmgr.msc
  4. Continue with the import process

Solution 2: Edit Group Policy on Domain Controller

It is common for PCs in office environment to be connected to a network domain. The default domain policy is to disallow users from adding their own trusted root CA certificate.

To get around this problem, use group policy management editor on your domain controller and find **Certificate Path Validation Settings**. Update the settings into following:

cert_import2

See this Microsoft article for more info.

Spring Basic Concept: Inversion of Control and Dependency Injection

If you’re a newbie just starting to learn Spring, or even if you’ve heard a lot about Spring but not sure what it was, Martin Fowler has an excellent article about this: Inversion of Control and Dependency Injection pattern.

It was a bit old (written in 2004) but every single conceptual discussion still applies today.

If any of the following keyword doesn’t make any sense to you, then I beg you spare 20 minutes of your time to read this excellent article:

  • Inversion of Control
  • Dependency Injection
  • Service Locator