MY Scribbling...

AWS Community Hero Masanori YAMAGUCHI の 雑なメモ

nginxを使ってgitlabとjenkinsをサブディレクトリでリバースプロキシする

git gitlab jenkins nginx

TL;DR

  • インフラの構成をコードで管理したいのでgitlabを使う
  • gitlabのレポジトリにPushしたコードは自動でテストをまわしたいのでjenkinsを使う
  • サーバ2台使うとリソースもったいたいので1台にまとめてnginxでリバースプロキシする
  • gitlabとjenkinsはアクセスするサブディレクトリで分ける
  • インスタンスはOpenStack上で動作

これらをセットアップしたときのメモです

構成

https://gyazo.com/d6e59c8dc38a472d6350cccd3ca96d00

Jenkinsのセットアップ

  • インストールはここを参照
  • サブディレクトリ /jenkins へのアクセスに対応するよう /etc/sysconfig/jenkinsを編集する(通常は http://xxx.xxx.xxx.xxx:8080 がjenkinsのURL)
    • JENKINS_ARGS に --prefix=/jenkins を渡す
## Type:        string
## Default:     ""
## ServiceRestart: jenkins
#
# Pass arbitrary arguments to Jenkins.
# Full option list: java -jar jenkins.war --help
#
JENKINS_ARGS="--prefix=/jenkins"
  • jenkinsを再起動する
# service jenkins restart
  • /var/log/jenkins/jenkins.log にエラーが出ていないことを確認
WARNING: Could not intialize the host network interface on nullbecause of an error: infra-ci-jenkins.novalocal
: infra-ci-jenkins.novalocal: unknown error
java.net.UnknownHostException: infra-ci-jenkins.novalocal: infra-ci-jenkins.novalocal: unknown error
        at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
        at javax.jmdns.impl.HostInfo.newHostInfo(HostInfo.java:75)
        at javax.jmdns.impl.JmDNSImpl.<init>(JmDNSImpl.java:407)
        at javax.jmdns.JmDNS.create(JmDNS.java:60)
        at hudson.DNSMultiCast$1.call(DNSMultiCast.java:32)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: infra-ci-jenkins.novalocal: unknown error
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
        at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
        ... 9 more

Gitlabのセットアップ

  • インストールはここを参照

  • /etc/gitlab/gitlab.rb を編集する

    • gitlabは、Rackサーバのunicornで動作しており、内部で同梱のnginxがリバースプロキシをしている。そのため、以下の箇所でgitlab同梱のunicornとnginxの設定を行っておく必要がある。
    • unicorn['listen'] を 127.0.0.1 にする
    • unicorn['port'] を 8888 にする(デフォルトの8080はJenkinsで使用している為)
    • nginx['listen_port'] を 10080 にする(デフォルトの80はGitlabとJenkinsをリバースプロキシするフロントのnginxで使用するため)
    • nginx['listen_https']を false にする(もしSSLにする場合はフロントのnginxで対応すれば良い)
##################
# GitLab Unicorn #
##################
## Tweak unicorn settings.

# unicorn['worker_timeout'] = 60
# unicorn['worker_processes'] = 2

## Advanced settings
unicorn['listen'] = '127.0.0.1'
unicorn['port'] = 8888
# unicorn['socket'] = '/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
# unicorn['pidfile'] = '/opt/gitlab/var/unicorn/unicorn.pid'
# unicorn['tcp_nopush'] = true
# unicorn['backlog_socket'] = 1024
# Make sure somaxconn is equal or higher then backlog_socket
# unicorn['somaxconn'] = 1024
# We do not recommend changing this setting
# unicorn['log_directory'] = "/var/log/gitlab/unicorn"

## Only change these settings if you understand well what they mean
## see https://about.gitlab.com/2015/06/05/how-gitlab-uses-unicorn-and-unicorn-worker-er/
## and https://github.com/kzk/unicorn-worker-killer
# unicorn['worker_memory_limit_min'] = "200*(1024**2)"
# unicorn['worker_memory_limit_max'] = "250*(1024**2)"

<snip>

################
# GitLab Nginx #
################
## see: https://gitlab.com/gitlab-org/omnibus-gitlab/e/629def0a7a26e7c2326566f0758d4a27857b52a3/doc/set    tings/nginx.md

# nginx['enable'] = true
# nginx['client_max_body_size'] = '250m'
# nginx['redirect_http_to_https'] = false
# nginx['redirect_http_to_https_port'] = 80
# nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/ca.crt" # Most root CA's are luded by default
# nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"
# nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"
# nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256"
# nginx['ssl_prefer_server_ciphers'] = "on"
# nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2" # recommended by https://raymii.orgtutorials/Strong_    SSL_Security_On_nginx.html & https://cipherli.st/
# nginx['ssl_session_cache'] = "builtin:1000  shared:SSL:10m" # recommended in http://nx.org/en/docs/ht    tp/ngx_http_ssl_module.html
# nginx['ssl_session_timeout'] = "5m" # default according to http://nginx.org/en/docs/p/ngx_http_ssl_mo    dule.html
# nginx['ssl_dhparam'] = nil # Path to dhparams.pem, eg. /etc/gitlab/ssl/dhparams.pem
# nginx['listen_addresses'] = ['*']
# nginx['listen_port'] = nil # override only if you use a reverse proxy: https://lab.com/gitlab-org/omn    ibus-gitlab/blob/master/doc/settings/nginx.md#setting-the-nx-listen-port
nginx['listen_port'] = 10080
# nginx['listen_https'] = nil # override only if your reverse proxy internally municates over HTTP: htt    ps://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/tings/nginx.md#supporting-proxied-ssl
nginx['listen_https'] = false
# nginx['custom_gitlab_server_config'] = "location ^~ /foo-namespace/bar-project/raw/  deny all;\n}\n"
# nginx['custom_nginx_config'] = "include /etc/nginx/conf.d/example.conf;"
# nginx['proxy_read_timeout'] = 300
# nginx['proxy_connect_timeout'] = 300
  • gitlabを再構成する
    • gitlabは構成管理にchefが使われている
    • 前述のgitlab.rbはそのためのAttributeファイルになっている
# gitlab-ctl reconfigure

gitlab-ctlでは構成されない範囲があるので以下は手動で各設定ファイルを編集していく必要がある

  • /opt/gitlab/embedded/service/gitlab-rails/config/application.rb 
# Relative url support
# Uncomment and customize the last line to run in a non-root path
# WARNING: We recommend creating a FQDN to host GitLab in a root path instead of 
# Note that following settings need to be changed for this to work.
# 1) In your application.rb file: config.relative_url_root = "/gitlab"
# 2) In your gitlab.yml file: relative_url_root: /gitlab
# 3) In your unicorn.rb: ENV['RAILS_RELATIVE_URL_ROOT'] = "/gitlab"
# 4) In ../gitlab-shell/config.yml: gitlab_url: "http://127.0.0.1/gitlab"
# 5) In lib/support/nginx/gitlab : do not use asset gzipping, remove block g with "location ~ ^    /(assets)/"
#
# To update the path, run: sudo -u git -H bundle exec rake assets:precompile NV=production
#
config.relative_url_root = "/gitlab"
# WARNING: See config/application.rb under "Relative url support" for the list of
# other files that need to be changed for relative url support
relative_url_root: /gitlab
  • /var/opt/gitlab/gitlab-rails/etc/unicorn.rb
    • ENV['RAILS_RELATIVE_URL_ROOT'] = "/gitlab" を最終行に追記する
ENV['RAILS_RELATIVE_URL_ROOT'] = "/gitlab"
gitlab_url: "http://127.0.0.1:8888/gitlab"
  • gitlabを再起動する
# gitlab-ctl restart
  • /var/log/gitlab/nginx/gitlab_error.log や /var/log/gitlab/unicorn/unicorn_stderr.logにエラーが出力されていないことを確認
I, [2016-01-12T00:50:57.143568 #26720]  INFO -- : Refreshing Gem list
E, [2016-01-12T00:51:11.272678 #26720] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use)
E, [2016-01-12T00:51:11.272884 #26720] ERROR -- : retrying in 0.5 seconds (4 tries left)
E, [2016-01-12T00:51:11.773372 #26720] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use)
E, [2016-01-12T00:51:11.774122 #26720] ERROR -- : retrying in 0.5 seconds (3 tries left)
E, [2016-01-12T00:51:12.274524 #26720] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use)
E, [2016-01-12T00:51:12.274673 #26720] ERROR -- : retrying in 0.5 seconds (2 tries left)
E, [2016-01-12T00:51:12.775115 #26720] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use)
E, [2016-01-12T00:51:12.775266 #26720] ERROR -- : retrying in 0.5 seconds (1 tries left)
E, [2016-01-12T00:51:13.275751 #26720] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use)
E, [2016-01-12T00:51:13.275970 #26720] ERROR -- : retrying in 0.5 seconds (0 tries left)
E, [2016-01-12T00:51:13.776510 #26720] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use)
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/socket_helper.rb:185:in `bind': Address already in use - bind(2) for 127.0.0.1:8080 (Errno::EADDRINUSE)
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/socket_helper.rb:185:in `new_tcp_server'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/socket_helper.rb:165:in `bind_listen'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:242:in `listen'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:809:in `block in bind_new_listeners!'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:809:in `each'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:809:in `bind_new_listeners!'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:138:in `start'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `load'
        from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `<main>'

nginxのセットアップ

  • nginxのインストールはここを参照

  • /etc/nginx/conf.d/ に以下のファイルを作成する

    • jenkins.conf
    • gitlab.conf
upstream jenkins_server {
 server 127.0.0.1:8080 fail_timeout=0;
}
upstream gitlab_server {
 server 127.0.0.1:8888 fail_timeout=0;
}
  • /etc/nginx.conf/conf.d/default.confを以下のように編集する(ここではdefault.confはファイル名を変更し、basic.confとしている)
server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/log/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # Jenkins
    location ~ /jenkins {
        proxy_read_timeout      300;
        proxy_connect_timeout   300;
        proxy_redirect          off;

        proxy_set_header        X-Forwarded-Proto       $scheme;
        proxy_set_header        Host                    $http_host;
        proxy_set_header        Host                    $host;
        proxy_set_header        X-Real-IP               $remote_addr;
        proxy_set_header        X-Forwarded-For         $proxy_add_x_forwarded_for;

        proxy_pass              http://jenkins_server;

        access_log              /var/log/nginx/jenkins_access.log;
        error_log               /var/log/nginx/jenkins_error.log;
    }
    # gitlab
    location ~ /gitlab {
        proxy_read_timeout      300;
        proxy_connect_timeout   300;
        proxy_redirect          off;

        proxy_set_header        X-Forwarded-Proto       $scheme;
        proxy_set_header        Host                    $http_host;
        proxy_set_header        Host                    $host;
        proxy_set_header        X-Real-IP               $remote_addr;
        proxy_set_header        X-Forwarded-For         $proxy_add_x_forwarded_for;

        proxy_pass              http://gitlab_server;

        access_log              /var/log/nginx/gitlab_access.log;
        error_log               /var/log/nginx/gitlab_error.log;
    }
}

  • /etc/nginx/conf.d/example_ssl.conf ファイルを削除する

  • nginxを再起動する

# chkconfig nginx on
# service nginx restart
  • /var/log/nginx/error.log、/var/log/nginx/jenkins_error.log、 /var/log/nginx/gitlab_error.logにエラーが出力されていないことを確認
[error] 1226#0: *225 connect() failed (111: Connection refused) while connecting to upstream, client: 172.24.4.254, server: localhost, request: "GET /gitlab HTTP/1.1", upstream: "http://127.0.0.1:8888/gitlab", host: "10.0.1.63:13580"

接続してみる

http://xxx.xxx.xxx.xxx/jenkins と http://xxx.xxx.xxx.xxx/gitlab にアクセスして、それぞれのサービスのトップ画面が表示されればOK

  • jenkins

https://gyazo.com/37d1e9eb68a5ff012a5c6a7880d1310e

  • gitlab

https://gyazo.com/0d07ef51fe5986e2ac9aaa2be3d431f6

done.