[RHEL7,COMMIT] ms/mm/list_lru.c: set bit in memcg shrinker bitmap on first list_lru item appearance

Submitted by Konstantin Khorenko on Sept. 5, 2018, 9:37 a.m.


Message ID 201809050937.w859bCKp010297@finist_ce7.work
State New
Series "Port "Improve shrink_slab() scalability" patchset"
Headers show

Commit Message

Konstantin Khorenko Sept. 5, 2018, 9:37 a.m.
The commit is pushed to "branch-rh7-3.10.0-862.11.6.vz7.71.x-ovz" and will appear at https://src.openvz.org/scm/ovz/vzkernel.git
after rh7-3.10.0-862.11.6.vz7.71.8
commit 24eb05575a04dfb377f16ee30a10c19e0a1a84f7
Author: Kirill Tkhai <ktkhai@virtuozzo.com>
Date:   Wed Sep 5 12:37:12 2018 +0300

    ms/mm/list_lru.c: set bit in memcg shrinker bitmap on first list_lru item appearance
    ms commit fae91d6d8be5
    Introduce set_shrinker_bit() function to set shrinker-related bit in
    memcg shrinker bitmap, and set the bit after the first item is added and
    in case of reparenting destroyed memcg's items.
    This will allow next patch to make shrinkers be called only, in case of
    they have charged objects at the moment, and to improve shrink_slab()
    [ktkhai@virtuozzo.com: v9]
    Link: http://lkml.kernel.org/r/153112557572.4097.17315791419810749985.stgit@localhost.localdomain
    Link: http://lkml.kernel.org/r/153063065671.1818.15914674956134687268.stgit@localhost.localdomain
    Signed-off-by: Kirill Tkhai <ktkhai@virtuozzo.com>
    Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
    Tested-by: Shakeel Butt <shakeelb@google.com>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
    Cc: Chris Wilson <chris@chris-wilson.co.uk>
    Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
    Cc: Guenter Roeck <linux@roeck-us.net>
    Cc: "Huang, Ying" <ying.huang@intel.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Josef Bacik <jbacik@fb.com>
    Cc: Li RongQing <lirongqing@baidu.com>
    Cc: Matthew Wilcox <willy@infradead.org>
    Cc: Matthias Kaehlcke <mka@chromium.org>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Cc: Michal Hocko <mhocko@kernel.org>
    Cc: Minchan Kim <minchan@kernel.org>
    Cc: Philippe Ombredanne <pombredanne@nexb.com>
    Cc: Roman Gushchin <guro@fb.com>
    Cc: Sahitya Tummala <stummala@codeaurora.org>
    Cc: Stephen Rothwell <sfr@canb.auug.org.au>
    Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Waiman Long <longman@redhat.com>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
    Signed-off-by: Kirill Tkhai <ktkhai@virtuozzo.com>
    Patchset description:
    Port "Improve shrink_slab() scalability" patchset
    This is backport of the patchset improving the performance
    of overcommited containers with many memcgs and mounts.
    The original set is in Linus' tree, and came into 4.19-rc1.
    Kirill Tkhai (12):
          mm: assign id to every memcg-aware shrinker
          mm/memcontrol.c: move up for_each_mem_cgroup{, _tree} defines
          mm, memcg: assign memcg-aware shrinkers bitmap to memcg
          fs: propagate shrinker::id to list_lru
          mm/list_lru.c: add memcg argument to list_lru_from_kmem()
          mm/list_lru: pass dst_memcg argument to memcg_drain_list_lru_node()
          mm/list_lru.c: pass lru argument to memcg_drain_list_lru_node()
          mm/list_lru.c: set bit in memcg shrinker bitmap on first list_lru item appearance
          mm/memcontrol.c: export mem_cgroup_is_root()
          mm/vmscan.c: iterate only over charged shrinkers during memcg shrink_slab()
          mm: add SHRINK_EMPTY shrinker methods return value
          mm/vmscan.c: clear shrinker bit if there are no objects related to memcg
    Vladimir Davydov (1):
          mm/vmscan.c: generalize shrink_slab() calls in shrink_node()
 include/linux/memcontrol.h |  4 ++++
 mm/list_lru.c              | 24 ++++++++++++++++++++++--
 mm/memcontrol.c            | 12 ++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)

Patch hide | download patch | download mbox

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 4d881adf12b5..6270653cfb91 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -633,6 +633,8 @@  static __always_inline struct mem_cgroup *mem_cgroup_from_kmem(void *ptr)
 	return __mem_cgroup_from_kmem(ptr);
 extern int memcg_expand_shrinker_maps(int new_id);
+extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg,
+				   int nid, int shrinker_id);
 #define for_each_memcg_cache_index(_idx)	\
 	for (; NULL; )
@@ -692,6 +694,8 @@  static inline struct mem_cgroup *mem_cgroup_from_kmem(void *ptr)
 	return NULL;
+static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg,
+					  int nid, int shrinker_id) { }
 #endif /* CONFIG_MEMCG_KMEM */
 #endif /* _LINUX_MEMCONTROL_H */
diff --git a/mm/list_lru.c b/mm/list_lru.c
index aca7dd7da162..21e12a8364ff 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -29,6 +29,11 @@  static void list_lru_unregister(struct list_lru *lru)
+static int lru_shrinker_id(struct list_lru *lru)
+	return lru->shrinker_id;
 static void list_lru_register(struct list_lru *lru)
@@ -37,6 +42,11 @@  static void list_lru_register(struct list_lru *lru)
 static void list_lru_unregister(struct list_lru *lru)
+static int lru_shrinker_id(struct list_lru *lru)
+	return -1;
 #endif /* CONFIG_MEMCG_KMEM */
@@ -113,13 +123,18 @@  bool list_lru_add(struct list_lru *lru, struct list_head *item)
 	int nid = page_to_nid(virt_to_page(item));
 	struct list_lru_node *nlru = &lru->node[nid];
+	struct mem_cgroup *memcg;
 	struct list_lru_one *l;
-	l = list_lru_from_kmem(nlru, item, NULL);
+	l = list_lru_from_kmem(nlru, item, &memcg);
 	if (list_empty(item)) {
 		list_add_tail(item, &l->list);
-		l->nr_items++;
+		/* Set shrinker bit if the first element was added */
+		if (!l->nr_items++)
+			memcg_set_shrinker_bit(memcg, nid,
+					       lru_shrinker_id(lru));
 		return true;
@@ -496,6 +511,7 @@  static void memcg_drain_list_lru_node(struct list_lru *lru, int nid,
 	struct list_lru_node *nlru = &lru->node[nid];
 	int dst_idx = memcg_cache_id(dst_memcg);
 	struct list_lru_one *src, *dst;
+	bool set;
 	 * Since list_lru_{add,del} may be called under an IRQ-safe lock,
@@ -507,9 +523,13 @@  static void memcg_drain_list_lru_node(struct list_lru *lru, int nid,
 	dst = list_lru_from_memcg_idx(nlru, dst_idx);
 	list_splice_init(&src->list, &dst->list);
+	set = (!dst->nr_items && src->nr_items);
 	dst->nr_items += src->nr_items;
+	if (set)
+		memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru));
 	src->nr_items = 0;
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 0ba355d1e47a..19d7d6553f59 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -824,6 +824,18 @@  int memcg_expand_shrinker_maps(int new_id)
 	return ret;
+void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id)
+	if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) {
+		struct memcg_shrinker_map *map;
+		rcu_read_lock();
+		map = rcu_dereference(memcg->info.nodeinfo[nid]->shrinker_map);
+		set_bit(shrinker_id, map->map);
+		rcu_read_unlock();
+	}
 static void disarm_kmem_keys(struct mem_cgroup *memcg)