Brillo开发: system/webservd代码分析 – webservd

brillo-m9-dev/brillo-m9-release版本release出来已经有一段时间了,之前一直在学习OpenWrt 15.05就暂停了一段时间。webservd是Brillo系统与外界交互的模块之一,我们之前一直使用curl命令去获取设备信息,进行认证以获取access token,进行wifi连接配置以及发送命令,都是通过webservd来传达的。webservd会通过external/libminihttp来建立一个简单的http/https server。

system/webservd(@ffc49cf)下面主要包含这两个模块:

  1. webservd
  2. libwebserv

这两个模块是完全独立的:libwebserv其实是weaved(system/weaved)的一下模块,是webservd的一个客户端。由于webservd需要通过dbus访问其他的服务如firewalld, keystore等。libwebserv与webservd也是通过dbus进行通信的。所以代码中还包含dbus binding相关的两个模块:

  1. libwebservd-client-internal: 由system/webservd/webservd/dbus_bindings/文件夹下的文件生成
  2. libwebserv-proxies-internal: 由system/webservd/libwebservd/dbus_bindings/文件夹下的文件生成
  • webservd – 文件结构
webservd/
├── Android.mk
├── config.cc
├── config.h
├── config_unittest.cc
├── dbus_bindings
│   ├── dbus-service-config.json
│   ├── org.chromium.WebServer.ProtocolHandler.dbus-xml
│   └── org.chromium.WebServer.Server.dbus-xml
├── dbus_protocol_handler.cc
├── dbus_protocol_handler.h
├── dbus_request_handler.cc
├── dbus_request_handler.h
├── encryptor.h
├── error_codes.cc
├── error_codes.h
├── etc
│   ├── dbus-1
│   │   └── org.chromium.WebServer.conf
│   └── init
│       └── webservd.conf
├── fake_encryptor.cc
├── fake_encryptor.h
├── firewalld_firewall.cc
├── firewalld_firewall.h
├── firewall_interface.h
├── keystore_encryptor.cc
├── keystore_encryptor.h
├── log_manager.cc
├── log_manager.h
├── log_manager_unittest.cc
├── main.cc
├── permission_broker_firewall.cc
├── permission_broker_firewall.h
├── protocol_handler.cc
├── protocol_handler.h
├── request.cc
├── request.h
├── request_handler_interface.h
├── server.cc
├── server.h
├── server_interface.h
├── temp_file_manager.cc
├── temp_file_manager.h
├── utils.cc
├── utils.h
├── webservd.rc
└── webservd_testrunner.cc

从文件名称可以看出webservd包含配置相关(config)、dbus进程间通信(dbus_xxx)、防火墙设置(firewall)、加密认证(keystore)、log管理、http/https协议请求处理、建立http/https服务、通过http/https协议上传文件管理(temp_file_manager)以及init配置文件。

其他文件:

  1. unit test相关的文件: config_unittest.cc, webservd_testrunner.cc
  2. 早期使用md5对RSA private key进行加密解密的代码: fake_encryptor.cc
  3. chromeos中设置firewall相关的代码: permission_broker_firewall.cc
  • webservd – 配置相关(config.cc)

主要包含:

  1. protocol handler name: “http”, “https”
  2. port number: default:  80(http), 443(https)
  3. tls(Transport Layer Security): https(true), http(false)
  4. interface name: 如eth0, wlan0
  5. RSA private key: 配置https service
  6. certificate:  配置https service
  7. certificate fingerprint
  8. debug: 通过命令行参数进行配置
  9. use ipv6: 通过命令行参数进行配置
  10. log directory
  11. default request timeout: 60 seconds

web server的默认配置如下:

// Default configuration for the web server.
const char kDefaultConfig[] = R"({
  "protocol_handlers": [
    {
      "name": "http",
      "port": 80,
      "use_tls": false
    },
    {
      "name": "https",
      "port": 443,
      "use_tls": true
    }
  ]
})";

在web server默认配置中,定义了两个protocol handler: http & https。web server配置也可通过命令行参数进行指定。

  • webservd – 启动流程

先看一下system/webservd/webservd/webservd.rc文件:

on post-fs-data
    mkdir /data/misc/webservd 0700 webserv webserv
    mkdir /data/misc/webservd/logs 0700 webserv webserv
    mkdir /data/misc/webservd/uploads 0700 webserv webserv

service webservd /system/bin/webservd
    class late_start
    user webserv
    group webserv dbus inet

init进行在post-fs-data阶段,会在/data/misc/webservd下面创建:

  1. logs – 存放log
  2. uploads – 通过web server上传的文件去先存放在这里

同时可以看出webservd以webserv的用户、webserv dbus inet的组运行。

然后就到了main.cc:

在main函数里,会先处理命令行参数:

  1. log_to_stderr: 是否将log输出到stderr,默认为false
  2. config_path: web server的配置文件,默认为空,使用default configuration
  3. debug: 是否在web请求中打印debug信息,默认为false
  4. ipv6:  是否要支持ipv6, 默认为true

再接着加载配置文件:

const char kDefaultConfigFilePath[] = "/etc/webservd/config";
// ...
int main(int argc, char* argv[]) {
  // ...
  base::FilePath default_file_path{kDefaultConfigFilePath};
  if (!FLAGS_config_path.empty()) {
    // In tests, we'll override the board specific and default configurations
    // with a test specific configuration.
    webservd::LoadConfigFromFile(base::FilePath{FLAGS_config_path}, &config);
  } else if (base::PathExists(default_file_path)) {
    // Some boards have a configuration they will want to use to override
    // our defaults.  Part of our interface is to look for this in a
    // standard location.
    CHECK(webservd::LoadConfigFromFile(default_file_path, &config));
  } else {
    webservd::LoadDefaultConfig(&config);
  }
  // ...
}

可以看到优先加载命令行指定的配置文件,再接着从/etc/webservd/config加载配置文件,最后才使用默认的配置。

再接着就是根据配置信息创建network interface socket,  network interface socket可以绑定到特定的network interface(如eth0, wlan0),如果不指定,就可以接收任意network interface的请求:

int CreateNetworkInterfaceSocket(const std::string& if_name) {
  // The following is basically the steps libmicrohttpd normally takes
  // when creating a new listening socket and binding it to a port.
  int socket_fd = socket(PF_INET6, SOCK_STREAM | SOCK_CLOEXEC, 0);
  if (socket_fd < 0 && errno == EINVAL)
    socket_fd = socket(PF_INET6, SOCK_STREAM, 0);
  if (socket_fd < 0) {
    PLOG(ERROR) << "Unable to create a listening socket";
    return -1;
  }

  // Now, specify that we want this socket to bind only to a particular
  // network interface. Note, this call requires root privileges, so this
  // should be done before the privileges are dropped.
  if (setsockopt(socket_fd, SOL_SOCKET, SO_BINDTODEVICE,
                 if_name.c_str(), if_name.size()) < 0) {
    PLOG(WARNING) << "Failed to bind socket (SO_BINDTODEVICE) to " << if_name;
    close(socket_fd);
    return -1;
  }
  return socket_fd;
}

最后执行brillo::Daemon::Run() @ system/webservd/webservd/main.cc:

-> main() @ system/webservd/webservd/main.cc
-> brillo::Daemon::Run() @ external/librillo/brillo/daemons/daemon.cc
-> brillo::DBusServiceDaemon::OnInit() @ external/libbrillo/brillo/daemons/dbus_daemon.cc
-> (anonymous namespace)::Daemon::RegisterDBusObjectsAsync() @ system/webservd/webservd/main.cc

在(anonymous namespace)::Daemon::RegisterDBusObjectsAsync()中:

-> 初始化log manager

-> 创建firewall service client

-> 创建webservd::Server对象

`    -> 创建dbus object

`    -> 创建default encryptor

`    -> 设置default request timeout

->执行webservd::RegisterAsync()

`    -> 初始化tls

`    -> 创建protocol handler

`    -> 注册firewall service online callback: webservd::FirewalldFirewall::WaitForServiceAsync()

`    -> export Server

`     -> export ProtocolHandler

  • webservd – log管理

由system/webservd/webservd/log_manager.cc可以看出最多可以有7个log文件,文件名以日期的形式命名,如2015-02-25.log,单个文件最大可以到1MB, 如果文件大于1MB, 那么会被重命名为日期+[a-z].log的形式,如2015-02-15-a.log的形式。

log记录的web客户端ip地址,服务器响应请求的时间,方法(GET, POST, PUT, PATCH),访问的url, http版本号,响应的状态码以及大小, 如:

127.0.0.1 - - [25/Feb/2015:03:29:12 -0800] "GET /test HTTP/1.1" 200 2326
  • webservd – 证书 ->@<-

建立https服务是需要证书的,brilloemulator使用的是自签名证书。创建证书的过程如下:

-> 创建X509 v3证书: CN=”Brillo device”, 有效期~5年

-> 创建RSA key pair: key length: 1024bits

-> 保存RSA private key到temp buffer

-> 创建EVP: The Digital EnVeloPe library @ openssl-1.0.1r/doc/ssleay.txt

-> EVP: assign RSA private key

-> EVP: 证书 + RSA public key

-> 证书签名

-> 将签名后的证书保存到/data/misc/webservd/certificate文件中

-> 使用key store对RSA private key进行加密

-> 将加密后的RSA private key保存到/data/misc/webservd/key文件中

调用的流程如下:

-> main() @ system/webservd/webservd/main.cc
-> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc
-> brillo::DBusServiceDaemon::OnInit() @ external/libbrillo/brillo/daemons/dbus_daemon.cc
-> (anonymous namespace)::Daemon::RegisterDBusObjectsAsync() @ system/webservd/webservd/main.cc
-> webservd::Server::RegisterAsync() @ system/webservd/webservd/server.cc
-> webservd::Server::InitTlsData() @ system/webservd/webservd/server.cc
  • webservd – 建立http/https服务
-> main() @ system/webservd/webservd/main.cc
-> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc
-> brillo::DBusServiceDaemon::OnInit() @ external/libbrillo/brillo/daemons/dbus_daemon.cc
-> (anonymous namespace)::Daemon::RegisterDBusObjectsAsync() @ system/webservd/webservd/main.cc
-> webservd::Server::RegisterAsync() @ system/webservd/webservd/server.cc
-> webservd::Server::CreateProtocolHandler() @ system/webservd/webservd/server.cc
-> webservd::ProtocolHandler::Start() @ system/webservd/webservd/protocol_handler.cc
-> webservd::ProtocolHandler::DoWork() @ system/webservd/webservd/protocol_handler.cc
  • webservd – 防火墙配置

建立web server时,必然需要开放端口供客户端访问,Brillo系统使用的网络防火墙由iptable来实现,默认的配置如下(由system/etc/init.firewall-setup.sh完成):

#!/system/bin/sh
#
# Set up default firewall rules.

BINPATH=/system/bin

# IPv4 only rules.
iptables_icmp_setup() {
  ${BINPATH}/iptables -A INPUT -p icmp -j ACCEPT -w
}

iptables_mdns_setup() {
  ${BINPATH}/iptables -A INPUT -p udp --destination 224.0.0.251 --dport 5353 -j ACCEPT -w
}


# IPv6 only rules.
ip6tables_icmp_setup() {
  ${BINPATH}/ip6tables -A INPUT -p ipv6-icmp -j ACCEPT -w

  # Allow all outbound ICMPv6 traffic.  This is important for things like
  # neighbor discovery and address negotiation.
  ${BINPATH}/ip6tables -A OUTPUT -p ipv6-icmp -j ACCEPT -w
}

ip6tables_mdns_setup() {
  ${BINPATH}/ip6tables -A INPUT -p udp --destination FF02::FB --dport 5353 -j ACCEPT -w
}


# Install all IPv4 and IPv6 rules.
for iptables in ip{,6}tables; do
  iptables_bin=${BINPATH}/${iptables}
  [ -x ${iptables_bin} ] || continue

  # Set default policy to DROP.
  ${iptables_bin} -P INPUT DROP -w
  ${iptables_bin} -P FORWARD DROP -w
  ${iptables_bin} -P OUTPUT DROP -w

  # Accept everything on the loopback.
  ${iptables_bin} -I INPUT -i lo -j ACCEPT -w
  ${iptables_bin} -I OUTPUT -o lo -j ACCEPT -w

  # Accept return traffic inbound.
  ${iptables_bin} -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -w

  # Accept icmp echo (NB: icmp echo ratelimiting is done by the kernel).
  ${iptables}_icmp_setup

  # Accept new and return traffic outbound.
  ${iptables_bin} -I OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT -w

  # Accept inbound mDNS traffic.
  ${iptables}_mdns_setup

  # Accept DHCP traffic (communicating as either client or server).
  ${iptables_bin} -I INPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT -w
  ${iptables_bin} -I OUTPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT -w
done


# Set completion property.
setprop firewall.init 1

使用iptables命令查看可能会更直观一些:

hzak@B85PRO:~$ adb shell iptables --list
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     udp  --  anywhere             anywhere             udp spts:bootps:bootpc dpts:bootps:bootpc
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:5555
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251          udp dpt:mdns

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp spts:bootps:bootpc dpts:bootps:bootpc
ACCEPT     all  --  anywhere             anywhere             state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
hzak@B85PRO:~$ adb shell ip6tables --list
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     udp      anywhere             anywhere             udp spts:bootps:bootpc dpts:bootps:bootpc
ACCEPT     all      anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     icmpv6    anywhere             anywhere            
ACCEPT     udp      anywhere             ff02::fb             udp dpt:mdns

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     udp      anywhere             anywhere             udp spts:bootps:bootpc dpts:bootps:bootpc
ACCEPT     all      anywhere             anywhere             state NEW,RELATED,ESTABLISHED
ACCEPT     all      anywhere             anywhere            
ACCEPT     icmpv6    anywhere             anywhere

由于注册过firewall service online callback, 所以当firewall service online的时候,会回调:

void Server::OnFirewallServiceOnline() {
  LOG(INFO) << "Firewall service is on-line. "
            << "Opening firewall for protocol handlers";                                                
  for (auto& handler_config : config_.protocol_handlers) {                                              
    VLOG(1) << "Firewall request: Protocol Handler = " << handler_config.name                           
            << ", Port = " << handler_config.port << ", Interface = "                                   
            << handler_config.interface_name;                                                           
    firewall_->PunchTcpHoleAsync(                                                                       
        handler_config.port,
        handler_config.interface_name,                                                                  
        base::Bind(&OnFirewallSuccess, handler_config.interface_name,                                   
                   handler_config.port),                                                                
        base::Bind(&IgnoreFirewallDBusMethodError));
  }
} 
void FirewalldFirewall::PunchTcpHoleAsync(
    uint16_t port,
    const std::string& interface_name,
    const base::Callback<void(bool)>& success_cb,
    const base::Callback<void(brillo::Error*)>& failure_cb) {
  proxy_->PunchTcpHoleAsync(port, interface_name, success_cb, failure_cb);
}

去更新防火墙的设置,如打开80端口(http)

  • webservd – Add request handler

Add request handler是由weaved (system/weaved)发起的,向webservd注册web API。调用的过程如下:

-> main() @ system/webservd/webservd/main.cc
-> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc
-> brillo::BaseMessageLoop::Run() @ external/libbrillo/brillo/daemons/daemon.cc
-> base::RunLoop::Run() @ external/libchrome/base/run_loop.cc
-> base::MessagePumpLibevent::Run() @ external/libchrome/bae/message_loop/message_pump_libevent.cc
-> base::MessageLoop::DoWork() @ external/libchrome/base/message_loop/message_loop.cc
-> base::MessageLoop::DeferOrRunPendingTask() @ external/libchrome/base/message_loop/message_loop.cc
-> base::MessageLoop::RunTask() @ external/libchrome/base/message/message_loop.cc
-> base::debug::TaskAnnotator::RunTask() @ external/libchrome/base/debug/task_annotator.cc
-> base::Callback<>::Run() @ external/libchrome/base/callback.h
-> dbus::Bus::ProcessAllIncomingDataIfAny() @ external/libchrome/dbus/bus.cc
-> dbus_connection_dispatch() @ external/dbus/dbus/dbus-connection.cc
-> _dbus_object_tree_dispatch_and_unlock() @ external/dbus/dbus/dbus-object-tree.c
-> dbus::ExportedObject::HandleMessage() @ external/libchrome/dbus/exported_object.cc
-> base::Callback<>::Run() @ external/libchrome/base/callback.h
-> base::internal::Invoker<>::Run() @ external/libchrome/base/bind_internal.h
-> base::internal::InvokeHelper<>::MakeItSo() @ external/libchrome/base/bind_internal.h
-> base::internal::RunnableAdapter<>::Run() @ external/libchrome/base/bind_internal.h
-> brillo::dbus_utils::SimpleDBusInterfaceMethodHandlerWithErrorAndMessage() @ external/libbrillo/brillo/dbus/dbus_oject.cc
-> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h
-> brillo::dbus_utils::DBusParamReader<>::InvokeHelper() @ external/libbrillo/brillo/dbus/dbus_param_reader.h
-> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h
-> brillo::dbus_utils::DBusParamReader<>::InvokeHelper() @ external/libbrillo/brillo/dbus/dbus_param_reader.h
-> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h
-> brillo::dbus_utils::DBusParamReader<>::InvokeHelper() @ external/libbrillo/brillo/dbus/dbus_param_reader.h
-> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h
-> brillo::dbus_utils::SimpleDBusInterfaceMethodHandlerWithErrorAndMessage<>::HandleMethod()
-> base::Callback<>::Run() @ external/libchrome/base/callback.h
-> base::internal::Invoke<>::Run() @ external/libchrome/base/bind_internal.h
-> base::internal::InvokeHelper<>::MakeItSo() @ external/libchrome/base/bind_internal.h
-> base::internal::RunnableAdapter<>::Run() @ external/libchrome/base/bind_internal.h
-> webservd::DBusProtocolHandler::AddRequestHandler() @ system/webservd/webservd/dbus_protocol_handler.cc
-> webservd::ProtocolHandler::AddRequestHandler() @ system/webservd/webservd/protocol_handler.cc

我们再看一下webservd::ProtocolHandler::AddRequestHandler()的实现:

std::string ProtocolHandler::AddRequestHandler(
    const std::string& url,
    const std::string& method,
    std::unique_ptr<RequestHandlerInterface> handler) {
  std::string handler_id = base::GenerateGUID();
  request_handlers_.emplace(handler_id,
                            HandlerMapEntry{url, method, std::move(handler)});
  return handler_id;
}
  • webservd – 处理请求(handle request)

当我们在客户端执行命令时,webservd就会将请求通过D-Bus转发给weaved(system/weaved)。如我们执行如下命令:

$ curl -H "Authorization: Privet anonymous" -k https://localhost:8443/privet/info

webservd内部调用的流程如下:

-> main() @ system/webservd/webservd/main.cc
-> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc
-> brillo::BaseMessageLoop::Run() @ external/libbrillo/brillo/message_loops/base_message_loop.cc
-> base::RunLoop() @ external/libchrome/base/run_loop.cc
-> base::MessagePumpLibevent::Run() @ external/libchrome/base/message_loop/message_pump_libevent.cc
-> base::MessageLoop::DoWork() @ external/libchrome/base/message_loop/message_loop.cc
-> base::MessageLoop::DeferOrRunPendingTask() @ external/libchrome/base/message_loop/message_loop.cc
-> base::MessageLoop:RunTask() @ external/libchrome/base/message_loop/message_loop.cc
-> base::debug::TaskAnnotator::RunTask() @ external/libchrome/base/debug/task_annotator.cc
-> base::Callback<void ()>::Run() @ external/libchrome/base/callback.h
-> webservd::ProtocolHandler::DoWork() @ system/webservd/webservd/protocol_handler.cc
-> MHD_run() @ external/libmicrohttpd/src/microhttpd/daemon.c
-> MHD_select() @ external/libmicrohttpd/src/microhttpd/daemon.c
-> MHD_run_from_select() @ external/libmicrohttpd/src/microhttpd/daemon.c
-> MHD_connection_handle_idle() @ external/libmicrohttpd/src/microhttpd/connection.c
-> call_connection_handler() @ external/libmicrohttpd/src/microhttpd/connection.c
-> webservd::ServerHelper::ConnectionHandler() @ system/webservd/webservd/protocol_handler.cc
-> webservd::Request::BeginRequestData() @ system/webservd/webservd/request.cc
-> webservd::ForwardRequestToHandler() @ system/webservd/webservd/request.cc
// Send the request over D-Bus and await the response

从这里可以看到webservd中用到了libmicrohttpd,Message loop和D-Bus。

  • webservd – 上传文件

如果上传的数据中需要使用到文件,可以以表单的形式上传文件到Brillo系统主机。上传文件的命令如下:

$ curl -H "Authorization: Privet anonymous" -X POST -F update=@configure -k https://localhost:8443/privet/info

这里,update是key, 而configure是value, 对应的是上传的文件名。

服务器缓存的文件名的格式如下(<guid>-<counter>):

B515AE30-1FA3-4ABA-80A5-8EB2A409AB73-1

当请求结束后,上传的文件会被删除。

发表评论

电子邮件地址不会被公开。 必填项已用*标注