Skip to main content

Offramp Deployment Guide

A general inline deployment uses rWeb and one or more RIOS filter devices to protect your edge routers from upstream attack traffic. However, there may be cases where this deployment model is insufficent for your network for any number of reasons, the most common one being that an inline model requires you to site a filter device at each upstream link requiring protection.

Choosing where to locate your mitigation capacity and diverting traffic there (commonly referred to as off-ramping) can be performed by adding RioRey Director to your deployment as a supplement to your rWeb and RIOS filter devices. Director also enables more complicated deployment scenarios, such as providing scrubbing capability to networks other than your own, though this guide will follow a basic offramp scenario. Other examples can be found in the examples section.

This guide is intended for anyone who wants to begin using Director. While familiarity with rWeb or having an existing rWeb and RIOS deployment would be ideal, it is not a requirement. In order to deploy Director, readers are assumed to have a high degree of familiarity with their own network design and offramp requirements.

This guide will walk you through the following:

  • Installing and configuring rWeb
  • Installing and configuring Director
  • Modifications to your network to support offramping

Network Design​

Prior to installing anything, you should review your current network to determine where you intend to locate any filtering capability (in most cases this will be a RIOS filter device) and where your redirection interactions will happen. In the most common case, Director expects to process incoming flow samples from your network devices (i.e. NetFlow or sFlow from your edge routers) and change your network routing by making BGP announcements to your network. Director also needs to be able to communicate with rWeb in order to ensure it has the same set of Groups and Zones.

You should, at a minimum, be able to answer the following questions before proceeding:

  1. What is your network address space, and is it within an ASN under your control?
  2. Where is the offramp going to happen (both physically and from a network perspective)?
  3. Under what network conditions do you want to divert your traffic?
  4. How are you sampling your traffic? Which network devices need to send samples, using which protocols?

rWeb Deployment​

If you don't already have an rWeb deployed, you will need to install one and do some initial configuration. If you already have an rWeb, skip the install step and review the remaining steps.

You should install the rWeb where it will have management plane access to both Director and any RIOS filter devices you intend on deploying (question #2 from above).

Director queries rWeb for Group and Zone configuration, that it then uses for network event detection and triggering actions. More details on the rWeb and Director relationship is available here.

  1. Install an rWeb according the initial install guide. If you're getting up and running quickly, a minimum VM installation (requirements as outlined in the guide) will be sufficient. Copy down the serial number of rWeb to send to support@riorey.com later, as the VM will come up with a one week temporary license sufficient for doing initial configuration.
  2. Configure the Groups and Zones rWeb will be using. The routes assigned to Zones should make sense for your prefix space as identified in question #1 above.
warning

Director strips out the 0.0.0.0/0 and ::/0 routes associated with the Default Zone. You should ensure that your prefixes (identified in question #1 above) are either directly added to the Default Zone's routing or are present in a different Zone, as Director does not know how to sample traffic that isn't explicitly configured as part of your network.

  1. If you expect to make public BGP announcements, we recommend making sure each Zone's prefixes are large enough to be publicly announced using the minimum prefix size accepted by most public BGP peers (/24 for IPv4, and /48 for IPv6). The mask value used for actual announcements can be configured in Director, but it won't stop actions from affecting multiple Zones if the chosen mask hits their routes (e.g. if two Zones each have a /28 route, a /24 announcement triggered by conditions for one could end up affecting the other). Deconflicting these potential subnet mismatches is a manual process that should be reviewed ahead of time.

Director Installation​

Once rWeb is installed and configured, you will need to install Director and configure it. Director needs to be installed within your network such that it can communicate with rWeb, the network devices that will be sending it traffic flow samples, and any BGP peers or network devices that will be directly performing traffic redirection. In most cases the latter will be simpler with BGP, but you can load custom scripts into Director that may do this directly on other routing devices.

You may want to review some of the Director deployment concepts to familiarize yourself prior to installation. When ready, install Director according to the initial install guide. Unlike rWeb, you will need to get a license directly from RioRey Support before you can use Director. You should only proceed with the remainder of this guide after Director has been licensed.

Initial Configuration​

After installing Director, you will need to perform initial configuration. The main configuration element for network behavior in Director is the configuration TOML. Each deployment requires a minimum set of configuration items, irrespective of your final network design:

  1. Add an rWeb configuration block, pointing at the rWeb you deployed in the previous section. This requires the url, user, and either the password or password-file keys to be filled out.
info

The rWeb configuration block has a default value of false for the ignore-cert setting. If you don't have a formal publicly signed SSL certificate for your rWeb, you may need to set this value to true to allow Director to connect to your rWeb.

  1. Add a flow sampling configuration block using the preferred protocol you identified in question #4 above. Within the flow protocol section (which will either be a NetFlow variant or sFlow block), specify a ports list of the destination UDP ports where flow samples will be sent to Director.
  2. If you intend to use any type of BGP redirection, you should add a BGP configuration block. Director will run an instance of BGP, identified by the router-id and asn keys (the latter of which should have been identified in question #1 above). Each BGP peer you intend to communicate with should have have its own BGP Neighbor block configured. Director will make and withdraw announcements to these peers, but otherwise won't propagate routes.

Once everything is configured in your TOML file, issue a director reload config command (over ssh, or if you are in the director CLI already just reload config) to start using your updated configuration. The reload command will attempt to immediately import the config file as your running configuration.

info

While on the normal command line, you have access to the director_validate command. You can issue this command in conjunction with a filepath to check that the designated file has correct Director configuration syntax. This checks the file but does not import new configuration, and can be helpful to draft and test new configuration prior to issuing a reload.

Once you've issued the reload, you should have the minimum amount of configuration on Director to begin observing and collecting network data. You can check a few things before the next steps:

  • In the Director CLI, issue a show config zone command. If your Director instance is talking to your rWeb correctly, you should see the Groups and Zones with their defined prefixes. This example has the Default Zone with a /24 prefix added, and one additional Group with its own child Zone and a separate /24 prefix:
# show config zone
ID  |         Name         | Group ID | Group Name | Workflows |     Subnets
0   | Default              | 0        | Default    |           | 203.0.113.0/24
1   | Test Zone            | 1        | Test Group |           | 198.51.100.0/24
info

In the Director CLI, if you need help looking up commands you can always hit the '?' key to show a list of available options or the expected value if you are mid-command.

  • If you configured a BGP peer, you can check that you've established a session by issuing gobgp neighbor from the command line:
[admin@director ~]$ gobgp neighbor
Peer           AS      Up/Down State       |#Received  Accepted
172.29.33.4 64514  0d 01:20:44 Establ      |        0         0

Data Collection​

In order to begin network monitoring, you'll need to configure your network devices to send flow samples to Director and verify that you are receiving them:

  1. Configure your edge network devices to send flow samples to Director. This is so Director can do out-of-band monitoring to check when it should trigger a configured workflow. The specifics of the configuration will depend on whether you are using NetFlow or sFlow, as well as the brand of device you are using, so refer to the instructions of the device manufacturer. If performing redirection for an external network, this may require additional set up, such as a network tunnel to facilitate sending external network samples to Director.
  2. Verify that the samples are being received using the show sample commands in the Director CLI. You should see sample counts going up over time, and Director should auto-calculate the incoming sample rate for a given source. If you configured a network device to send samples but don't see its IP reported, check the network path between it and Director to ensure flow samples are being routed correctly and not dropped by intervening security devices. You should also check that you configured Director to listen to the correct sample types and ports.
# show sample counts
Sample Source IP |    Type    | Count
172.29.33.4      | sflow      | 4659
172.30.254.101   | netflow_v9 | 595764

# show sample rates
        Sample Source IP          |  Type   | Sample Rate (1 in N Packets)
172.29.33.4, sub-agent 100000_0_3 | sflow   | 400
172.30.254.101                    | netflow | 1024
  1. After samples are being received, you can use the show traffic-current CLI commands to verify that Director reports roughly correct current traffic conditions:
info

Director CLI traffic reporting operates on recorded packet samples, which are written every 30 seconds, so "roughly correct" should be a comparison with this in mind. This does not affect Director triggers, which activate based on live up-to-the-second rate calculations.

warning

In order to see traffic reporting, you must have additional prefixes configured in rWeb, per the rWeb section above and the discussion here. If you have rWeb with just the Default Zone's initial configuration, it won't be able to aggregate traffic data.

# show traffic-current
Querying Current bits_incoming for all by total
Rows processed:  100.0% [====================================================================>] 5292795/5292795 eta [00:00]
Top 10 results:
Sum(bits_incoming)
228.2 Mbps

# show traffic-current packets
Querying Current packets_incoming for all by total
Rows processed:  100.0% [====================================================================>] 5292795/5292795 eta [00:00]
Top 10 results:
Sum(packets_incoming)
540.5 kpps

# show traffic-current outgoing
Querying Current bits_outgoing for all by total
Rows processed:  100.0% [====================================================================>] 5297438/5297438 eta [00:00]
Top 10 results:
Sum(bits_outgoing)
15.2 Mbps

# show traffic-current outgoing packets
Querying Current packets_outgoing for all by total
Rows processed:  100.0% [====================================================================>] 5297438/5297438 eta [00:00]
Top 10 results:
Sum(packets_outgoing)
27.6 kpps

Design Workflows​

Once the main configuration options are set and you are monitoring your network, you can begin designing your workflows. Director uses a few key concepts to structure behavior: mappings, workflows, and actions.

  • A mapping associates Groups and/or Zones with a set of workflows.
  • A workflow has a set of trigger thresholds, and the set of actions that will be triggered when those thresholds are met.
  • The set of actions are what will ultimately trigger redirection, either through BGP announcements (normal BGP or FlowSpec), or through custom scripts. Custom scripts can also be used to perform functions other than redirection (e.g. send notifications).

The triggers and actions for a workflow should eventually implement the logic you identified in Network Design #3 above. Starting off, however, you should keep your triggers and actions empty, and instead begin scaffolding out your workflows:

[workflows.Test_Workflow]
actions=[]
trigger=[]

With an empty workflow, you can map the desired set of Groups or Zones against the workflow in a mapping declaration:

[[mapping]]
group-id-recursive = 1
workflows = ["Test_Workflow"]

In the Director CLI, issuing config reload followed by show config zone should show the workflow did indeed get mapped:

# show config zone
ID  |         Name         | Group ID | Group Name |   Workflows   |     Subnets
0   | Default              | 0        | Default    |               | 203.0.113.0/24
1   | Test Zone            | 1        | Test Group | Test_Workflow | 198.51.100.0/24

At this point the goal is to make sure any desired workflow for your network has empty actions and trigger declarations, and that your relevant Groups and/or Zones are present in the mapping declaration for each workflow.

Design Actions​

The specifics of your redirection action(s) should be identified in Network Design #1 and #2 above. A basic scrubbing example is available as a starting point, along with more complex examples. The examples omit the common configuration elements shown in initial configuration. At this point, you want to ensure that your actions have the desired effects in your network through configuration, validation, and testing:

  1. Configure your network to enable your desired actions. For example, adding any required VRFs or alternate routes for traffic to reach the mitigation appliances. These should be the configured paths used by your redirection actions. These paths may require additional physical modification if you are installing RIOS filter devices over these links for traffic cleaning purposes.
  2. If using BGP, ensure Director is set up as a peer for your existing BGP deployment (anybody you wrote a bgp.neighbor block for should also have Director configured on their end). You may have already checked this using gobgp neighbors as described in initial configuration. If using custom scripts for redirection (instead of BGP), you will need to load these on Director.
  3. Do some dry run testing of your actions. The best way is to define your desired actions and associate them in a temporary workflow. For example, if we wanted to test BGP redirection for our previously defined Test Zone (this should issue a BGP redirect for the Zone's /24 prefix):
info

The "always-on" trigger activates the workflow regardless of network conditions, behaving as if a per-zone trigger was activated.

[[mapping]]
workflows = ["action-test"]
zone-ids = [1]

[workflows.action-test]
actions=["test-redirect"]
[[workflows.action-test.trigger]]
type = "always-on"

[actions.test-redirect]
type="bgp-v4"
destination="detected"
next-hop="192.0.2.0"

As defined actions get tested and verified, they can be left defined but removed from the temporary workflow:

[[mapping]]
workflows = ["action-test"]
zone-ids = [1]

# The workflow is still active but doing nothing
[workflows.action-test]
actions=[]
[[workflows.action-test.trigger]]
type = "always-on"

# Now that this is tested let's rename it
[actions.redirect]
type="bgp-v4"
destination="detected"
next-hop="192.0.2.0"

Design Workflow Triggers​

Once you have your workflows and a reliable set of actions defined, the final remaining step is to identify useful triggers for when they should activate for a given workflow. This will basically be the conditions identified under Network Design #3, but this should be tested and implemented against your live network conditions.

To that end, it's recommended that you collect as much data as possible (Director will maintain a week's worth of traffic sample data) and then check your traffic conditions using show traffic queries. After you query traffic history, define your triggers in your workflows with the desired actions, and then repeat this process as necessary. There are a few broad things to consider when defining your triggers:

Per-host versus Per-zone​

Director can initiate actions based on either a per-host or per-zone basis. Per-host triggers are a good default value to start with, as host behavior typically should be more constrained and this is less sensitive to changes in subnet utilization. If you are observing regular attacks over a range of IPs, it might make more sense to use per-zone triggers for one or more zones that encapsulate that IP range. The Director CLI show traffic functions make it easy to query both, using the for clause to limit the query to the Groups and Zones mapped to the workflow.

To illustrate this, say our current draft mapping looks like this:

[[mapping]]
group-id-recursive = 0
workflows = [
        "Mitigate",
        "Emergency",
]

We can query against the workflow, where these queries will check every individual Zone for their peak traffic (due to group-id-recursive = 0), as well as the highest observed individual peak, in bits per second:

# show traffic for workflow Mitigate per zone
Querying max(bits_incoming) for workflow 'Mitigate' by total calculated per zone
Rows processed:  100.0% [==================================================================>] 95771931/95771931 eta [00:00]
Top 10 results:
Max(bits_incoming)
40.6 Mbps

# show traffic for workflow Mitigate per host
Querying max(bits_incoming) for workflow 'Mitigate' by total calculated per host
Rows processed:  100.0% [==================================================================>] 95771931/95771931 eta [00:00]
Top 10 results:
Max(bits_incoming)
25.4 Mbps

Triggers by Traffic Type​

The broadest traffic type that can be used for triggers will be either raw packet rate or raw bit rate. The queries shown in the previous section provided peak values for incoming bit rates. These can be the most useful traffic types initially, but depending on your needs, you may want to specialize some of your triggers and actions to handle specific conditions. For example, a general type of network attack is a SYN flood, so you may want to redirect hosts observing high incoming TCP SYN packet rates through a RIOS device for filtering. You can query those behaviors as well:

# show traffic for workflow Mitigate tcp-syn packets per host
Querying max(tcp_syn_packets_incoming) for workflow 'Mitigate' by total calculated per host
Rows processed:  100.0% [================================================================>] 180002525/180002525 eta [00:00]
Top 10 results:
Max(tcp_syn_packets_incoming)
15.2 kpps

Aggressive vs Conservative Triggers​

The examples so far show how to query traffic conditions for hosts and zones and various types, but the point of these queries is to examine observed traffic for where to actually set your trigger thresholds. By default the queries will perform peak calculations across all available data. With that data, you should also consider the following:

  • How severe is the action being associated with the trigger? You may not want to set up a blackhole announcement for something right around your peak values, as you may end up too aggressively blackholing your traffic. You may instead want to set up a redirect to a RIOS filter device.
  • Traffic can generally be sent through RIOS at will. By design RIOS filter devices are meant to be at a network edge handling all observed traffic, so there are no downsides to sending spikes of traffic through them at any trigger level.
  • If you want to be conservative with a given trigger, setting it to roughly 3x your weekly observed peak value means that it won't trigger prematurely, but only activate under problematic network conditions. It's easier to walk a trigger down than scale it up.

High Value RIOS Zone Configuration​

When redirecting traffic through RIOS filter devices, you may want to spend some additional effort for high-value or special needs traffic to ensure you don't see false positives and accidentally drop good traffic. Ideally, you've already identified and collected this traffic into its own specific Zone (or Zones, possibly under their own Group). Once the Zone is initially set up, we recommend the following to review the mitigation profile for the Zone:

  • Set the Zone to monitor mode. This will ensure the RIOS device reports attack traffic that it sees but won't actually drop anything.
  • Configure a static workflow mapped to this Zone with an always-on trigger to redirect its traffic through RIOS. An example static redirect shows how to do this, though the example is diverting only UDP DNS traffic (you'd omit the port and protocol declarations to send all traffic through).
  • Review the reported victims in rWeb to look for false positives. If false postives are observed, you may want to reduce filter sensitivities or make other algorithm changes as needed.

Operational Adjustments​

With Director fully deployed and many operational workflows, occassionally you may run into situations where you need to make adjustments. Here are some common scenarios and possible ways to address them.

Triggers too sensitive​

  • If your trigger is across broad categories like all packet or bit rates, you may want to switch it to a more specific packet type, such as tcp-syn.
  • Different Groups or Zones might have dramatically different traffic profiles. If the traffic patterns in one are triggering actions that end up affecting the other, you might want to adjust the mapping and move a Zone to a different workflow and trigger entirely.
  • Similarly, traffic to hosts within a Zone might have different traffic profiles that are affecting your triggers. For both Director and rWeb in general, it's generally better to cluster hosts by their network behavior, so you may want to create a new Zone if a subset of the hosts have dramatically different network behaviors.

Need to Stop Actions for a Zone​

If you find yourself needing to immediately stop all actions for a particular Zone, the fastest way is to create a new mapping entry for that Zone with no worfklows. In the configuration file, later mapping entries overwrite earlier ones. So you can append an empty entry, issue a director reload config, and immediately stop any actions triggered against that Zone. You still need to be careful in case you have address overlaps between the Zone you're trying to stop actions against, and a different Zone which has active actions that affect that overlap.

Need to Stop a Specific Action​

If you have a particular action that is misbehaving (such as a broken BGP setup or a bad custom script), you may want to stop just that action specifically. In that case, you can modify the configuration to add:

[actions.Misbehaving-Script]
type = "command-v4"
# This script is acting up, set it to no-op
no-op = true

This will immediately stop all instances of that action of executing, regardless of which workflow they are associated with.