Restricting website access using Cloudflare Firewall

Firewall is one of the main features of Cloudflare. This network-level protection is especially useful when your website has a lot of visitors and has higher potential for being abused or attacked, either by bots or real human-being. Anyways, it doesn’t hurt to use cloudflare firewall by any means.

This firewall feature is enabled by default and will protect your domain and subdomains which are using Cloudflare proxy (not DNS only). This firewall function comes with all Cloudflare plans, including the free plan. However, if you upgrade to a paid plan, you can use more firewall features such as Web Application Firewall (WAF) and bots protection. In this article we will talk about some very basic Cloudflare Firewall under Firewall Rules.

For this tutorial, we will try to set up a firewall that will restrict all subdomains in our account for the whole internet except for those who are given access to. This is useful for example, you have many internal sites under the subdomain of your main domain. You want the main domain still publicly accessible, but only allow all internal subdomains only for your employees. In this scenario, we will use:

  1. IP address based restriction
  2. Cookie-based restriction

Most companies have static IP addresses for their office network and VPN, so we can simply whitelist the IP address and allow all employees to connect to our internal websites. However, this won’t work if you want to share access to external clients. Another problem might also occur when your employees are working from home and can’t use VPN, and that will make them get blocked. This is why we recommend combining cookie-based rules for your firewall. Let’s get started.

Prerequisites:

  1. Domain and / or subdomains managed with cloudflare with proxy enabled
  2. Your office network / VPN static public IP

Steps:

  1. Go to cloudflare firewall – Firewall Rules, then click on Create firewall rule
  2. Configure the following. Rule name is something that you can easily memorize. When incoming requests match… is the restriction that you need to define. Then… is what you want Cloudflare does when it matches your rule.
    In this example, we prepare the following firewall expression to make you get started easily. Simply replace yourdomain.com with your domain and 111.111.111..111 with your office / VPN IP address. See step 4 for the rule and how to insert it into cloudflare.
  3. Click on Edit expression, then insert this rule (remember to replace the domain and IP address).


(not ip.src in {111.111.111.111} and not http.host in {“yourdomain.com”} and not http.cookie contains “cloudflare-firewall=lkj23894knlf943rowIUsd9frKskjswIPVwMs2aa”)


This expression consists of three parts:

  1. IP part (111.111.111.111)
  2. Domain (yourdomain.com)
  3. Cookie key definition (cloudflare-firewall=lkj23894knlf943rowIUsd9frKskjswIPVwMs2aa). You will need to use this cookie to visit the website if you are not using a whitelisted IP address.
  1. Select what action to take. We recommend using Challenge (captcha) to test if you are in a production environment. Just to be safe.
  2. Click save, and your rule should be deployed immediately. If not, click on the toggle to enable it.

Now it is time to test in action. Browse to your subdomains with and without your whitelisted IP address. Browse to it with and without your defined cookies. If you configure it correctly and you are not using whitelisted IP nor the cookies, then you will get blocked like me, good luck!

P.S: To set cookie when browsing, you can simply execute the below command on your browser console:


document.cookie = “cloudflare-firewall=lkj23894knlf943rowIUsd9frKskjswIPVwMs2aa”;

Or, you can also use browser extensions for setting the cookie like this one, but there are many other alternatives.

Capturing real visitor IP behind Cloudflare proxy + Nginx

By default, nginx will log visitor IP addresses in its access log. This is mostly useful for debugging purposes when you have some problem with your server. Also it’s often useful for mitigating bot attacks, hacking attempts, or other security-related issues.

Cloudflare is another DNS-level tool that can be used to speed up your website performance, increase security of your website, and many others. This time we will focus on cloudflare’s basic function called Cloudflare proxy. 

In the picture above you can see whether the DNS record is proxied through cloudflare. When you enable proxy, all visitor requests won’t directly hit your webserver and the actual server IP will be hidden. In contrast to traditional DNS (or, in Cloudflare they call it “DNS only”), we can usually know what is the actual server IP which serves a specific domain. This cloudflare proxy is good for both security and speed. Good for security because the actual server IP will remain hidden for the whole world, and good for speed because Cloudflare will acts as a “proxy” that will serve HTTP requests from cloudflare cache (if any). 

However, this proxy thing can sometimes cause a problem. One of them is this that we’re discussing in this article. If we enable proxy, then it’s gonna be the cloudflare proxy servers that will actually hit our server, not the real customer. That way, nginx will record cloudflare’s IP addresses instead of the visitor’s. To address this problem, we can apply some simple nginx configuration so that it will record the real customer IP. We assume that you already have a website running on nginx webserver and you have registered your domain on cloudflare.

Prerequisites:

  1. Nginx with http_realip_module module (should come by default in standard nginx installation)

Steps:

  1. Edit your site’s nginx config:
    vim /etc/nginx/sites-enabled/your-nginx-config-here
  2. Add the following lines. This list of IPs might change. Please check here for an up-to-date list.

set_real_ip_from 103.21.244.0/22;

set_real_ip_from 103.22.200.0/22;

set_real_ip_from 103.31.4.0/22;

set_real_ip_from 104.16.0.0/13;

set_real_ip_from 104.24.0.0/14;

set_real_ip_from 108.162.192.0/18;

set_real_ip_from 131.0.72.0/22;

set_real_ip_from 141.101.64.0/18;

set_real_ip_from 162.158.0.0/15;

set_real_ip_from 172.64.0.0/13;

set_real_ip_from 173.245.48.0/20;

set_real_ip_from 188.114.96.0/20;

set_real_ip_from 190.93.240.0/20;

set_real_ip_from 197.234.240.0/22;

set_real_ip_from 198.41.128.0/17;

set_real_ip_from 2400:cb00::/32;

set_real_ip_from 2606:4700::/32;

set_real_ip_from 2803:f800::/32;

set_real_ip_from 2405:b500::/32;

set_real_ip_from 2405:8100::/32;

set_real_ip_from 2c0f:f248::/32;

set_real_ip_from 2a06:98c0::/29;

#use any of the following two

real_ip_header CF-Connecting-IP;

#real_ip_header X-Forwarded-For;

  1. Save & quit
  2. Reload nginx
    sudo systemctl reload nginx.service

That’s it! Now you should be able to see the actual visitor IP under your nginx’s access log. Let us know if it works!

Magento – Allow Specific IP Address on Maintenance Mode

Usually when redeploying new code or introducing critical changes to the server environment. It is common to restrict access to end-users for avoiding errors caused by end-users using an invalid or incorrect environment state.

Magento allows developers to set maintenance mode, redirecting all users to the application 503 page. 

bin/magento maintenance:enable

Also, the developers have the advantage of whitelisting one or more IP addresses, allowing them to test any change in the server without being worried about any end-user accessing the server at the same the server is being updated. 

For example, the following command will enable maintenance mode for all end users except for 192.0.2.20 and 192.0.2.21:

$bin/magento maintenance:enable –ip=192.0.2.20 –ip=192.0.2.21

Enabled maintenance mode

Set exempt IP-addresses: 192.0.2.20, 192.0.2.21

The command above hopefully will be a helpful tool in order to reduce server downtime, the same for maintenance and development time.

Magento PhpStorm Plugin not Working

If you just installed the newest PhpStorm version or just upgraded from a previous version you might notice that the Magento PhpStorm plugin is not working; not showing any plugin-related menu nor allowing you to autogenerate code, reducing your job efficiency.

You are also sure that you followed the correct steps for installing the plugin but it still not working, even worse, the PHP -> Frameworks section is also missing from the IDE.

What happened?  

It turns out that since PhpStorm 2021 1.1, the PHP frameworks section is independently listed at the top on File->Settings, and for some reason, this detail is also missing from the official PhpStorm documentation:

That section will look exactly the same as previous PhpStorm versions, just click on Enable Magento Integration and select the Magento installation path.

After clicking OK the IDE will reindex and the menus will start displaying all the cool features from the plugin, speeding up your code development.

Happy coding!

Duplicate Entry error in Magento – How to deal with it?

It is inevitable that your Magento version; hosted by your server or cloud environment; will be updated at some point, due to being pushed by software or hardware upgrades, and there might be some rare scenarios that after an upgrade you will get the following error after executing some regular operations on both frontend and backend:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry

Normally, when getting this error, the common solution is to check the database for the following:

  • If you know which tables are involved in the executed process; check that the increment property is correctly set in the corresponding column.
  • Check for any unique property set in a column that is not allowing to insert/update new data.

However, the above solutions require a deeper understanding of Magento’s database structure, and changing the database manually might be risky, it might lead to data loss if you are not careful.

Luckily, there are scenarios that might not require such dangerous troubleshooting. For example, if the error is caused when saving/updating a product with Tier Price you would need to search for the module processing the update and check if the store scope is explicitly set by using calling setCurrentStore(), similar to this:

$this->_storeManager->setCurrentStore(0);

$this->_productRepository->save($product);

And just make the following modification for using the deprecated getResource()->save() function instead of the recommended Product Repository:

$this->_storeManager->setCurrentStore(0);

$product->getResource()->save($product);

//$this->_productRepository->save($product);

By using the deprecated getResource()->save() function Magento will overcome this issue and save the product normally, but of course, it would reduce performance a little bit by not using the ProductRepository.

At the moment of writing this article, there is no official plugin that addresses this issue on the ProductRepository, but at least it will allow you to avoid direct changes in the database. 

Another situation that would trigger the Duplicate Entry error is when trying the save a CMS page, but fortunately, Magento already released the patch MDVA-33606 for version 2.4.1:

MDVA-33606: solves the issue where the users get Unique constraint violation found error when saving a CMS page assigned to hierarchy tree. 

Finally, for applying a patch you must follow the instructions for your specific Magento version. The present article will not cover the different methods for applying a patch. If your Magento version is lower than 2.4.1 you can still check out the official Magento patch information and implement a similar solution to your version.

Happy coding!

A Guide for Upgrading Magento Cloud

Upgrading Magento Cloud is very simple but it will require you to read a lot of documentation for understanding the correct action plan; so for this reason this article aims to facilitate information on the upgrading procedure in order to help you in making a decision whether your environment needs to upgraded or not.  

Step 1: DB backup: 

Be sure to create all the necessary backups in order to easily roll back the environment in case the upgrading somehow failed or some data is missing after the upgrade.

Step 2: Upgrade Magento Services: 

Update the services that interact between them in the environment. Before upgrade please check the correct services version configuration for your environment.

Step 3: Upgrade the PHP version: 

Update the PHP version installed in the environment through docker. Before upgrade please check the correct version according to the version of the services to be installed or that are already running in your environment.

Step 4: Upgrade the Magento version: 

Update the Magento version installed in the environment through docker. Before upgrade please check the correct version according to the version of the services to be installed or that are already running in your environment.

Step 5: Apply patches (If needed): 

Create all the necessary patches or just apply the patches provided by Magento (just in case it is necessary for your environment.

Please remember that this is a general guide for upgrading your Magento Cloud environment; however, there might be some specific situations that might require specific modifications; but this document might give you a good starting point.

Initial Steps for Upgrading a Magento Cloud Environment

By design, the Magento Cloud solution eases the way for upgrading any particular environment. However, some precautions need to be taken in case the upgrading results are not as the owner and/or administrator expected.

Step 1: Check environment health

Before starting any upgrade process, the owner/administrator must check the current status of the environment to be upgraded. This will discard any issue related to the upgrade process.

Step 2: Take snapshot

The easiest way to create a snapshot is by going to the Cloud Interface, then selecting the Master environment and finally clicking on snapshots.

By doing this, the snapshot will save the current state for all environments/branches. Note: depending on the package, besides the Master environment, some other environment might allow taking snapshots.

Another way to take a snapshot is by executing the magento cloud CLI mgc snapshot:create.

$ mgc snapshot:create

Step 3: Database backup 

The fastest way to backup the database is by executing the by executing the magento cloud CLI mgc db:dump, this command will automatically download the database content of the selected environment.

$ mgc db:dump

Although, the upgrading process is very simple; by following these simple recommendations, the upgrading process might have more resources for troubleshooting in case of dealing with unexpected setbacks.

Ubuntu – Black Screen during Login

On some rare occasions, after an upgrade; due to compatibility issues between hardware/software components,  when trying to logging you might only see a nasty black screen, or the OS will bring you back to the logins screen; meaning that you are not able to log in at all into Ubuntu.

In most cases, the solution requires direct access to shell and manually make some adjustments. 

Open shell view

You can boot on shell root (no need to login) or press CTRL+ALT+F5 to switch to shell view.

Login and initial prompt

Create a test account for discarding configuration issues.

Creating a temporary account will help to discard any configuration issue in the home directory. The following commands will create a new temporary account for testing purposes:

sudo adduser temporary

sudo systemctl reboot

If you are not able to log in with the new temporary account it means that the issue is within your original account’s home folder; so for this reason, the old configuration files need to be deleted in order to allow a normal login:

rm -rf  ~/.config

rm -rf ~/.local

rm -rf ~/.cache

rm -rf ~/.nvidia-settings-rc

rm ~/.nv

sudo systemctl reboot

If after rebooting the issue persists, please go back to shell view and continue troubleshooting.

Reinstall the login manager

After an upgrade, some core UI related components might be causing a conflict, so it is recommended to reinstall the GNOME Display Manager and the desktop environment by running the following commands:

sudo apt install –reinstall gdm3 ubuntu-desktop gnome-shell

sudo systemctl reboot

If after rebooting the issue persists, please go back to shell view and continue troubleshooting.

Reinstall NVIDIA Driver

A recent NVIDIA update might also be causing the login issues, so for this reason it would need to be reinstalled. To remove the NVIDIA driver, run the following commands:

sudo apt purge ~nnvidia

sudo apt autoremove

sudo apt clean

Removing NVIDIA

After the NVIDIA driver has been removed, reinstall it by executing the following commands:

sudo apt update

sudo apt full-upgrade

sudo ubuntu-drivers autoinstall

sudo systemctl reboot

In most cases, after following these steps, Ubuntu should be able to normally log into Ubuntu; if the problem persists you might need to consider to re-install your OS or look more deeply into any hardware issue.

Upgrading Magento version on Magento Cloud

Upgrading the Magento version for any particular environment in Magento Cloud could vary depending on the version to be installed and other running Magento services.

Before the upgrade please check the recommended configuration for the specific Magento version to be installed; by doing this, you would reduce the redeployment time and decrease any risk of unexpected installation errors.

You can check here and here for more information on how to select the correct configuration for your environment.

Upgrading the Magento version is as follows: 

  1. By using CLI, change to your local Magento root directory and set the upgrade version using the version constraint syntax:

composer require “magento/magento-cloud-metapackage”:”>=CURRENT_VERSION <NEXT_VERSION” –no-update

For example, when upgrading from version 2.3.7 to version 2.3.8, the execution should as follows:

composer require “magento/magento-cloud-metapackage”:”>=2.3.7 <2.3.8″ –no-update

  1. Manually edit the composer.json file for modifying the PHPUnit accordingly with the PHP version, just as indicated in the following table
PHPUnit versionPHP version
9>= 7.3
8>= 7.2
77.1 ~ 7.4
67.0 ~ 7.2

In case you are not performing unit or integration tests you might just delete the entries on the require-dev section and leave all blank:

“require-dev”: {

},

  1. For avoiding conflicts with previous packages you might need to remove the composer.lock file for forcing regeneration of this file:

  1. Update the project.
composer update –with dependencies
  1. Add, commit, and push code changes.

git add -A

git commit -m “Upgrade Magento version”

git push origin <branch-name>

  1. Wait for deployment to complete.
  1. Verify the upgrade in the environment by using SSH to log in and check the version.
php bin/magento –version

After verifying the Magento version, the environment is ready to be used. Also, do not forget to create backups of your database, code, and media files before upgrading the Magento version in the environment.

Ubuntu – Booting in Root Shell

Usually, when Ubuntu is installed on a desktop PC, the booting process will go through the GRUB menu, where you can select the OS to boot and/or select a different booting option. However, there are situations that might urge you to use a root shell, such as booting errors, driver installation, hardware configuration, etc.

Supposing that the Ubuntu version is 14.04, or later when the GRUB menu is fully loaded you can select Advanced options for Ubuntu.

enter image description here

After that, a list of available kernels will be available  for booting in Recovery mode 

enter image description here

Then, after pressing on any of the recovery options the screen will display the Recovery Menu, where you can choose Drop to a root shell prompt.

enter image description here

At this point, a terminal will be open, and there is no need to sudo for any task; however, 

if you require writing access to all files,  you must mount the filesystem by executing the following command:

mount -o rw,remount /

After finalizing executing all required activities, you can execute exit for returning to the GRUB menu.

exit

After this, every change made in the root console will be applied immediately in the next booting. 

Happy coding!